[cn1:13986] progress: initialized event flag to: 3
[cn1:13986] progress: initialized yield_when_idle to: false
[cn1:13986] progress: initialized num users to: 0
[cn1:13986] progress: initialized poll rate to: 23940000
[cn1:13986] progress: event_users_increment setting count to 1
[cn1:13986] mca: base: components_register: registering ess components
[cn1:13986] mca: base: components_register: found loaded component env
[cn1:13986] mca: base: components_register: component env has no register or open function
[cn1:13986] mca: base: components_register: found loaded component hnp
[cn1:13986] mca: base: components_register: component hnp has no register or open function
[cn1:13986] mca: base: components_register: found loaded component pmi
[cn1:13986] mca: base: components_register: component pmi has no register or open function
[cn1:13986] mca: base: components_register: found loaded component singleton
[cn1:13986] mca: base: components_register: component singleton register function successful
[cn1:13986] mca: base: components_register: found loaded component slurm
[cn1:13986] mca: base: components_register: component slurm has no register or open function
[cn1:13986] mca: base: components_register: found loaded component tool
[cn1:13986] mca: base: components_register: component tool has no register or open function
[cn1:13986] mca: base: components_open: opening ess components
[cn1:13986] mca: base: components_open: found loaded component env
[cn1:13986] mca: base: components_open: component env open function successful
[cn1:13986] mca: base: components_open: found loaded component hnp
[cn1:13986] mca: base: components_open: component hnp open function successful
[cn1:13986] mca: base: components_open: found loaded component pmi
[cn1:13986] mca: base: components_open: component pmi open function successful
[cn1:13986] mca: base: components_open: found loaded component singleton
[cn1:13986] mca: base: components_open: component singleton open function successful
[cn1:13986] mca: base: components_open: found loaded component slurm
[cn1:13986] mca: base: components_open: component slurm open function successful
[cn1:13986] mca: base: components_open: found loaded component tool
[cn1:13986] mca: base: components_open: component tool open function successful
[cn1:13986] mca:base:select: Auto-selecting ess components
[cn1:13986] mca:base:select:(  ess) Querying component [env]
[cn1:13986] mca:base:select:(  ess) Skipping component [env]. Query failed to return a module
[cn1:13986] mca:base:select:(  ess) Querying component [hnp]
[cn1:13986] mca:base:select:(  ess) Query of component [hnp] set priority to 100
[cn1:13986] mca:base:select:(  ess) Querying component [pmi]
[cn1:13986] mca:base:select:(  ess) Skipping component [pmi]. Query failed to return a module
[cn1:13986] mca:base:select:(  ess) Querying component [singleton]
[cn1:13986] mca:base:select:(  ess) Skipping component [singleton]. Query failed to return a module
[cn1:13986] mca:base:select:(  ess) Querying component [slurm]
[cn1:13986] mca:base:select:(  ess) Skipping component [slurm]. Query failed to return a module
[cn1:13986] mca:base:select:(  ess) Querying component [tool]
[cn1:13986] mca:base:select:(  ess) Skipping component [tool]. Query failed to return a module
[cn1:13986] mca:base:select:(  ess) Selected component [hnp]
[cn1:13986] mca: base: close: component env closed
[cn1:13986] mca: base: close: unloading component env
[cn1:13986] mca: base: close: component pmi closed
[cn1:13986] mca: base: close: unloading component pmi
[cn1:13986] mca: base: close: component singleton closed
[cn1:13986] mca: base: close: unloading component singleton
[cn1:13986] mca: base: close: component slurm closed
[cn1:13986] mca: base: close: unloading component slurm
[cn1:13986] mca: base: close: component tool closed
[cn1:13986] mca: base: close: unloading component tool
[cn1:13986] [[INVALID],INVALID] Topology Info:
[cn1:13986] Type: Machine Number of child objects: 2
	Name=NULL
	total=25100516KB
	DMIProductName=S5520UR
	DMIProductVersion=....................
	DMIBoardVendor="Intel Corporation"
	DMIBoardName=S5520UR
	DMIBoardVersion=E22554-751
	DMIBoardAssetTag=....................
	DMIChassisVendor=...............................
	DMIChassisType=23
	DMIChassisVersion=..................
	DMIChassisAssetTag=....................
	DMIBIOSVendor="Intel Corp."
	DMIBIOSVersion=S5500.86B.01.00.0054.092820101104
	DMIBIOSDate=09/28/2010
	DMISysVendor="Intel Corporation"
	Backend=Linux
	LinuxCgroup=/
	OSName=Linux
	OSRelease=2.6.32-431.3.1.el6.x86_64
	OSVersion="#1 SMP Fri Jan 3 21:39:27 UTC 2014"
	Architecture=x86_64
	Cpuset:  0x000000ff
	Online:  0x000000ff
	Allowed: 0x000000ff
	Bind CPU proc:   TRUE
	Bind CPU thread: TRUE
	Bind MEM proc:   FALSE
	Bind MEM thread: FALSE
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12517604KB
		total=12517604KB
		Cpuset:  0x00000055
		Online:  0x00000055
		Allowed: 0x00000055
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x00000055
			Online:  0x00000055
			Allowed: 0x00000055
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x00000055
				Online:  0x00000055
				Allowed: 0x00000055
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000001
					Online:  0x00000001
					Allowed: 0x00000001
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000001
						Online:  0x00000001
						Allowed: 0x00000001
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000001
							Online:  0x00000001
							Allowed: 0x00000001
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000001
								Online:  0x00000001
								Allowed: 0x00000001
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000004
					Online:  0x00000004
					Allowed: 0x00000004
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000004
						Online:  0x00000004
						Allowed: 0x00000004
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000004
							Online:  0x00000004
							Allowed: 0x00000004
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000004
								Online:  0x00000004
								Allowed: 0x00000004
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000010
					Online:  0x00000010
					Allowed: 0x00000010
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000010
						Online:  0x00000010
						Allowed: 0x00000010
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000010
							Online:  0x00000010
							Allowed: 0x00000010
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000010
								Online:  0x00000010
								Allowed: 0x00000010
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000040
					Online:  0x00000040
					Allowed: 0x00000040
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000040
						Online:  0x00000040
						Allowed: 0x00000040
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000040
							Online:  0x00000040
							Allowed: 0x00000040
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000040
								Online:  0x00000040
								Allowed: 0x00000040
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12582912KB
		total=12582912KB
		Cpuset:  0x000000aa
		Online:  0x000000aa
		Allowed: 0x000000aa
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x000000aa
			Online:  0x000000aa
			Allowed: 0x000000aa
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x000000aa
				Online:  0x000000aa
				Allowed: 0x000000aa
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000002
					Online:  0x00000002
					Allowed: 0x00000002
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000002
						Online:  0x00000002
						Allowed: 0x00000002
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000002
							Online:  0x00000002
							Allowed: 0x00000002
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000002
								Online:  0x00000002
								Allowed: 0x00000002
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000008
					Online:  0x00000008
					Allowed: 0x00000008
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000008
						Online:  0x00000008
						Allowed: 0x00000008
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000008
							Online:  0x00000008
							Allowed: 0x00000008
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000008
								Online:  0x00000008
								Allowed: 0x00000008
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000020
					Online:  0x00000020
					Allowed: 0x00000020
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000020
						Online:  0x00000020
						Allowed: 0x00000020
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000020
							Online:  0x00000020
							Allowed: 0x00000020
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000020
								Online:  0x00000020
								Allowed: 0x00000020
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000080
					Online:  0x00000080
					Allowed: 0x00000080
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000080
						Online:  0x00000080
						Allowed: 0x00000080
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000080
							Online:  0x00000080
							Allowed: 0x00000080
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000080
								Online:  0x00000080
								Allowed: 0x00000080
[cn1:13986] mca: base: components_register: registering plm components
[cn1:13986] mca: base: components_register: found loaded component isolated
[cn1:13986] mca: base: components_register: component isolated has no register or open function
[cn1:13986] mca: base: components_register: found loaded component rsh
[cn1:13986] mca: base: components_register: component rsh register function successful
[cn1:13986] mca: base: components_register: found loaded component slurm
[cn1:13986] mca: base: components_register: component slurm register function successful
[cn1:13986] mca: base: components_open: opening plm components
[cn1:13986] mca: base: components_open: found loaded component isolated
[cn1:13986] mca: base: components_open: component isolated open function successful
[cn1:13986] mca: base: components_open: found loaded component rsh
[cn1:13986] mca: base: components_open: component rsh open function successful
[cn1:13986] mca: base: components_open: found loaded component slurm
[cn1:13986] mca: base: components_open: component slurm open function successful
[cn1:13986] mca:base:select: Auto-selecting plm components
[cn1:13986] mca:base:select:(  plm) Querying component [isolated]
[cn1:13986] mca:base:select:(  plm) Query of component [isolated] set priority to 0
[cn1:13986] mca:base:select:(  plm) Querying component [rsh]
[cn1:13986] [[INVALID],INVALID] plm:rsh_lookup on agent ssh : rsh path NULL
[cn1:13986] mca:base:select:(  plm) Query of component [rsh] set priority to 10
[cn1:13986] mca:base:select:(  plm) Querying component [slurm]
[cn1:13986] [[INVALID],INVALID] plm:slurm: available for selection
[cn1:13986] mca:base:select:(  plm) Query of component [slurm] set priority to 75
[cn1:13986] mca:base:select:(  plm) Selected component [slurm]
[cn1:13986] mca: base: close: component isolated closed
[cn1:13986] mca: base: close: unloading component isolated
[cn1:13986] mca: base: close: component rsh closed
[cn1:13986] mca: base: close: unloading component rsh
[cn1:13986] plm:base:set_hnp_name: initial bias 13986 nodename hash 1752458269
[cn1:13986] plm:base:set_hnp_name: final jobfam 15051
[cn1:13986] mca: base: components_register: registering oob components
[cn1:13986] mca: base: components_register: found loaded component tcp
[cn1:13986] mca: base: components_register: component tcp register function successful
[cn1:13986] mca: base: components_open: opening oob components
[cn1:13986] mca: base: components_open: found loaded component tcp
[cn1:13986] mca: base: components_open: component tcp open function successful
[cn1:13986] mca:oob:select: checking available component tcp
[cn1:13986] mca:oob:select: Querying component [tcp]
[cn1:13986] oob:tcp: component_available called
[cn1:13986] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn1:13986] [[15051,0],0] oob:tcp:init rejecting loopback interface lo
[cn1:13986] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn1:13986] [[15051,0],0] oob:tcp:init creating module for V4 address on interface eth0
[cn1:13986] [[15051,0],0] creating OOB-TCP module for interface eth0
[cn1:13986] [[15051,0],0] oob:tcp:init adding 192.168.1.1 to our list of V4 connections
[cn1:13986] WORKING INTERFACE 3 KERNEL INDEX 3 FAMILY: V4
[cn1:13986] [[15051,0],0] oob:tcp:init creating module for V4 address on interface eth1
[cn1:13986] [[15051,0],0] creating OOB-TCP module for interface eth1
[cn1:13986] [[15051,0],0] oob:tcp:init adding 192.168.10.2 to our list of V4 connections
[cn1:13986] [[15051,0],0] TCP STARTUP
[cn1:13986] [[15051,0],0] attempting to bind to IPv4 port 0
[cn1:13986] [[15051,0],0] assigned IPv4 port 37478
[cn1:13986] mca:oob:select: Adding component to end
[cn1:13986] mca:oob:select: Found 1 active transports
[cn1:13986] mca: base: components_register: registering rml components
[cn1:13986] mca: base: components_register: found loaded component oob
[cn1:13986] mca: base: components_register: component oob has no register or open function
[cn1:13986] mca: base: components_open: opening rml components
[cn1:13986] mca: base: components_open: found loaded component oob
[cn1:13986] mca: base: components_open: component oob open function successful
[cn1:13986] orte_rml_base_select: initializing rml component oob
[cn1:13986] mca: base: components_register: registering routed components
[cn1:13986] mca: base: components_register: found loaded component binomial
[cn1:13986] mca: base: components_register: component binomial has no register or open function
[cn1:13986] mca: base: components_register: found loaded component debruijn
[cn1:13986] mca: base: components_register: component debruijn has no register or open function
[cn1:13986] mca: base: components_register: found loaded component direct
[cn1:13986] mca: base: components_register: component direct has no register or open function
[cn1:13986] mca: base: components_register: found loaded component radix
[cn1:13986] mca: base: components_register: component radix register function successful
[cn1:13986] mca: base: components_open: opening routed components
[cn1:13986] mca: base: components_open: found loaded component binomial
[cn1:13986] mca: base: components_open: found loaded component debruijn
[cn1:13986] mca: base: components_open: found loaded component direct
[cn1:13986] mca: base: components_open: found loaded component radix
[cn1:13986] mca:base:select: Auto-selecting routed components
[cn1:13986] mca:base:select:(routed) Querying component [binomial]
[cn1:13986] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn1:13986] mca:base:select:(routed) Querying component [debruijn]
[cn1:13986] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn1:13986] mca:base:select:(routed) Querying component [direct]
[cn1:13986] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn1:13986] mca:base:select:(routed) Querying component [radix]
[cn1:13986] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn1:13986] mca:base:select:(routed) Selected component [binomial]
[cn1:13986] mca: base: close: unloading component debruijn
[cn1:13986] mca: base: close: unloading component direct
[cn1:13986] mca: base: close: unloading component radix
[cn1:13986] orte_routed_base_select: initializing selected component binomial
[cn1:13986] mca: base: components_register: registering dstore components
[cn1:13986] mca: base: components_register: found loaded component hash
[cn1:13986] mca: base: components_register: component hash register function successful
[cn1:13986] mca: base: components_open: opening dstore components
[cn1:13986] mca: base: components_open: found loaded component hash
[cn1:13986] mca:dstore:select: checking available component hash
[cn1:13986] mca:dstore:select: Querying component [hash]
[cn1:13986] mca: base: components_register: registering grpcomm components
[cn1:13986] mca: base: components_register: found loaded component bad
[cn1:13986] mca: base: components_register: component bad has no register or open function
[cn1:13986] mca: base: components_register: found loaded component pmi
[cn1:13986] mca: base: components_register: component pmi register function successful
[cn1:13986] mca: base: components_open: opening grpcomm components
[cn1:13986] mca: base: components_open: found loaded component bad
[cn1:13986] mca: base: components_open: component bad open function successful
[cn1:13986] mca: base: components_open: found loaded component pmi
[cn1:13986] mca: base: components_open: component pmi open function successful
[cn1:13986] mca:base:select: Auto-selecting grpcomm components
[cn1:13986] mca:base:select:(grpcomm) Querying component [bad]
[cn1:13986] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn1:13986] mca:base:select:(grpcomm) Querying component [pmi]
[cn1:13986] mca:base:select:(grpcomm) Skipping component [pmi]. Query failed to return a module
[cn1:13986] mca:base:select:(grpcomm) Selected component [bad]
[cn1:13986] mca: base: close: component pmi closed
[cn1:13986] mca: base: close: unloading component pmi
[cn1:13986] [[15051,0],0] grpcomm:base:receive start comm
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 15
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 32
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 33
[cn1:13986] [[15051,0],0] plm:base:receive start comm
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 5
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 10
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 12
[cn1:13986] mca: base: components_register: registering ras components
[cn1:13986] mca: base: components_register: found loaded component loadleveler
[cn1:13986] mca: base: components_register: component loadleveler register function successful
[cn1:13986] mca: base: components_register: found loaded component simulator
[cn1:13986] mca: base: components_register: component simulator register function successful
[cn1:13986] mca: base: components_register: found loaded component slurm
[cn1:13986] mca: base: components_register: component slurm register function successful
[cn1:13986] mca: base: components_open: opening ras components
[cn1:13986] mca: base: components_open: found loaded component loadleveler
[cn1:13986] mca: base: components_open: component loadleveler open function successful
[cn1:13986] mca: base: components_open: found loaded component simulator
[cn1:13986] mca: base: components_open: found loaded component slurm
[cn1:13986] mca: base: components_open: component slurm open function successful
[cn1:13986] mca:base:select: Auto-selecting ras components
[cn1:13986] mca:base:select:(  ras) Querying component [loadleveler]
[cn1:13986] [[15051,0],0] ras:loadleveler: NOT available for selection
[cn1:13986] mca:base:select:(  ras) Skipping component [loadleveler]. Query failed to return a module
[cn1:13986] mca:base:select:(  ras) Querying component [simulator]
[cn1:13986] mca:base:select:(  ras) Skipping component [simulator]. Query failed to return a module
[cn1:13986] mca:base:select:(  ras) Querying component [slurm]
[cn1:13986] [[15051,0],0] ras:slurm: available for selection
[cn1:13986] mca:base:select:(  ras) Query of component [slurm] set priority to 50
[cn1:13986] mca:base:select:(  ras) Selected component [slurm]
[cn1:13986] mca: base: close: unloading component loadleveler
[cn1:13986] mca: base: close: unloading component simulator
[cn1:13986] mca: base: components_register: registering rmaps components
[cn1:13986] mca: base: components_register: found loaded component lama
[cn1:13986] mca:rmaps:lama: Priority   0
[cn1:13986] mca:rmaps:lama: Map   : NULL
[cn1:13986] mca:rmaps:lama: Bind  : NULL
[cn1:13986] mca:rmaps:lama: MPPR  : NULL
[cn1:13986] mca:rmaps:lama: Order : NULL
[cn1:13986] mca: base: components_register: component lama register function successful
[cn1:13986] mca: base: components_register: found loaded component mindist
[cn1:13986] mca: base: components_register: component mindist register function successful
[cn1:13986] mca: base: components_register: found loaded component ppr
[cn1:13986] mca: base: components_register: component ppr register function successful
[cn1:13986] mca: base: components_register: found loaded component rank_file
[cn1:13986] mca: base: components_register: component rank_file register function successful
[cn1:13986] mca: base: components_register: found loaded component resilient
[cn1:13986] mca: base: components_register: component resilient register function successful
[cn1:13986] mca: base: components_register: found loaded component round_robin
[cn1:13986] mca: base: components_register: component round_robin register function successful
[cn1:13986] mca: base: components_register: found loaded component seq
[cn1:13986] mca: base: components_register: component seq register function successful
[cn1:13986] mca: base: components_register: found loaded component staged
[cn1:13986] mca: base: components_register: component staged has no register or open function
[cn1:13986] mca: base: components_open: opening rmaps components
[cn1:13986] mca: base: components_open: found loaded component lama
[cn1:13986] mca: base: components_open: found loaded component mindist
[cn1:13986] mca: base: components_open: component mindist open function successful
[cn1:13986] mca: base: components_open: found loaded component ppr
[cn1:13986] mca: base: components_open: component ppr open function successful
[cn1:13986] mca: base: components_open: found loaded component rank_file
[cn1:13986] mca: base: components_open: component rank_file open function successful
[cn1:13986] mca: base: components_open: found loaded component resilient
[cn1:13986] mca: base: components_open: component resilient open function successful
[cn1:13986] mca: base: components_open: found loaded component round_robin
[cn1:13986] mca: base: components_open: component round_robin open function successful
[cn1:13986] mca: base: components_open: found loaded component seq
[cn1:13986] mca: base: components_open: component seq open function successful
[cn1:13986] mca: base: components_open: found loaded component staged
[cn1:13986] mca: base: components_open: component staged open function successful
[cn1:13986] mca:rmaps:select: checking available component lama
[cn1:13986] mca:rmaps:select: Querying component [lama]
[cn1:13986] mca:rmaps:select: checking available component mindist
[cn1:13986] mca:rmaps:select: Querying component [mindist]
[cn1:13986] mca:rmaps:select: checking available component ppr
[cn1:13986] mca:rmaps:select: Querying component [ppr]
[cn1:13986] mca:rmaps:select: checking available component rank_file
[cn1:13986] mca:rmaps:select: Querying component [rank_file]
[cn1:13986] mca:rmaps:select: checking available component resilient
[cn1:13986] mca:rmaps:select: Querying component [resilient]
[cn1:13986] mca:rmaps:select: checking available component round_robin
[cn1:13986] mca:rmaps:select: Querying component [round_robin]
[cn1:13986] mca:rmaps:select: checking available component seq
[cn1:13986] mca:rmaps:select: Querying component [seq]
[cn1:13986] mca:rmaps:select: checking available component staged
[cn1:13986] mca:rmaps:select: Querying component [staged]
[cn1:13986] [[15051,0],0]: Final mapper priorities
[cn1:13986] 	Mapper: ppr Priority: 90
[cn1:13986] 	Mapper: seq Priority: 60
[cn1:13986] 	Mapper: resilient Priority: 40
[cn1:13986] 	Mapper: mindist Priority: 20
[cn1:13986] 	Mapper: round_robin Priority: 10
[cn1:13986] 	Mapper: staged Priority: 5
[cn1:13986] 	Mapper: lama Priority: 0
[cn1:13986] 	Mapper: rank_file Priority: 0
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 34
[cn1:13986] [[15051,0],0] setting up session dir with
	tmpdir: UNDEF
	host cn1
[cn1:13986] procdir: /tmp/openmpi-sessions-artpol@cn1_0/15051/0/0
[cn1:13986] jobdir: /tmp/openmpi-sessions-artpol@cn1_0/15051/0
[cn1:13986] top: openmpi-sessions-artpol@cn1_0
[cn1:13986] tmp: /tmp
[cn1:13986] sess_dir_cleanup: job session dir does not exist
[cn1:13986] procdir: /tmp/openmpi-sessions-artpol@cn1_0/15051/0/0
[cn1:13986] jobdir: /tmp/openmpi-sessions-artpol@cn1_0/15051/0
[cn1:13986] top: openmpi-sessions-artpol@cn1_0
[cn1:13986] tmp: /tmp
[cn1:13986] [[15051,0],0] writing contact file /tmp/openmpi-sessions-artpol@cn1_0/15051/contact.txt
[cn1:13986] [[15051,0],0] wrote contact file
[cn1:13986] [[15051,0],0] routed_binomial: init routes for HNP job [15051,0]
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 2
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 21
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 22
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 45
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 46
[cn1:13986] progress: progress_set_yield_when_idle to false
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 1
[cn1:13986] [[15051,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 27
[cn1:13986] [[15051,0],0] Monitoring debugger attach fifo /tmp/openmpi-sessions-artpol@cn1_0/15051/0/debugger_attach_fifo
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 15 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 32 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 33 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 5 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 10 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 12 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 34 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 2 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 21 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 22 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 45 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 46 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 1 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] posting recv
[cn1:13986] [[15051,0],0] posting persistent recv on tag 27 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] plm:base:setup_job
[cn1:13986] [[15051,0],0] ras:base:allocate
[cn1:13986] [[15051,0],0] ras:slurm:allocate:discover: checking nodelist: cn[1-5]
[cn1:13986] [[15051,0],0] ras:slurm:allocate:discover: parse range 1-5 (2)
[cn1:13986] [[15051,0],0] ras:slurm:allocate:discover: adding node cn1 (2 slots)
[cn1:13986] [[15051,0],0] ras:slurm:allocate:discover: adding node cn2 (2 slots)
[cn1:13986] [[15051,0],0] ras:slurm:allocate:discover: adding node cn3 (2 slots)
[cn1:13986] [[15051,0],0] ras:slurm:allocate:discover: adding node cn4 (2 slots)
[cn1:13986] [[15051,0],0] ras:slurm:allocate:discover: adding node cn5 (2 slots)
[cn1:13986] [[15051,0],0] ras:slurm:allocate: success
[cn1:13986] [[15051,0],0] ras:base:node_insert inserting 5 nodes
[cn1:13986] [[15051,0],0] ras:base:node_insert updating HNP [cn1] info to 2 slots
[cn1:13986] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn1:13986] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn1:13986] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn1:13986] [[15051,0],0] ras:base:node_insert node cn2 slots 2
[cn1:13986] [[15051,0],0] ras:base:node_insert node cn3 slots 2
[cn1:13986] [[15051,0],0] ras:base:node_insert node cn4 slots 2
[cn1:13986] [[15051,0],0] ras:base:node_insert node cn5 slots 2
[cn1:13986] [[15051,0],0] plm:slurm: LAUNCH DAEMONS CALLED
[cn1:13986] [[15051,0],0] plm:base:setup_vm
[cn1:13986] [[15051,0],0] plm:base:setup_vm creating map
[cn1:13986] [[15051,0],0] plm:base:setup_vm add new daemon [[15051,0],1]
[cn1:13986] [[15051,0],0] plm:base:setup_vm assigning new daemon [[15051,0],1] to node cn2
[cn1:13986] [[15051,0],0] plm:base:setup_vm add new daemon [[15051,0],2]
[cn1:13986] [[15051,0],0] plm:base:setup_vm assigning new daemon [[15051,0],2] to node cn3
[cn1:13986] [[15051,0],0] plm:base:setup_vm add new daemon [[15051,0],3]
[cn1:13986] [[15051,0],0] plm:base:setup_vm assigning new daemon [[15051,0],3] to node cn4
[cn1:13986] [[15051,0],0] plm:base:setup_vm add new daemon [[15051,0],4]
[cn1:13986] [[15051,0],0] plm:base:setup_vm assigning new daemon [[15051,0],4] to node cn5
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 0 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial 0 found child 1
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 1 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 0
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 1
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 1 parent 0 me 1 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial 1 found child 3
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 3 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 0
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 1
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 1 parent 0 me 3 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 1
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 3
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 3 parent 1 me 3 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 1
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 1
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 0
[cn1:13986] [[15051,0],0] routed:binomial 0 found child 2
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 2 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 0
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 1
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 1 parent 0 me 2 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 1
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 3
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 3 parent 1 me 2 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 3
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 7
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 5
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 2
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 2 parent 0 me 2 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 0
[cn1:13986] [[15051,0],0] routed:binomial 0 found child 4
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 0
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 1
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 1 parent 0 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 1
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 3
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 3 parent 1 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 3
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 7
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 5
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 2
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 2 parent 0 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 2
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 6
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 4
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 4 parent 0 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 0
[cn1:13986] [[15051,0],0]: parent 0 num_children 3
[cn1:13986] [[15051,0],0]: 	child 1
[cn1:13986] [[15051,0],0]: 		relation 3
[cn1:13986] [[15051,0],0]: 	child 2
[cn1:13986] [[15051,0],0]: 	child 4
[cn1:13986] [[15051,0],0] plm:slurm: launching on nodes cn2,cn3,cn4,cn5
[cn1:13986] [[15051,0],0] plm:slurm: final top-level argv:
	srun --ntasks-per-node=1 --kill-on-bad-exit --cpu_bind=none --nodes=4 --nodelist=cn2,cn3,cn4,cn5 --ntasks=4 orted --debug --debug-daemons -mca orte_ess_jobid 986382336 -mca orte_ess_vpid 1 -mca orte_ess_num_procs 5 -mca orte_hnp_uri "986382336.0;tcp://192.168.1.1,192.168.10.2:37478"
Daemon was launched on cn5 - beginning to initialize
Daemon was launched on cn2 - beginning to initialize
Daemon was launched on cn4 - beginning to initialize
Daemon was launched on cn3 - beginning to initialize
[cn5:13991] progress: initialized event flag to: 3
[cn5:13991] progress: initialized yield_when_idle to: false
[cn5:13991] progress: initialized num users to: 0
[cn5:13991] progress: initialized poll rate to: 23940000
[cn5:13991] progress: event_users_increment setting count to 1
[cn5:13991] mca: base: components_register: registering ess components
[cn5:13991] mca: base: components_register: found loaded component env
[cn5:13991] mca: base: components_register: component env has no register or open function
[cn5:13991] mca: base: components_register: found loaded component hnp
[cn5:13991] mca: base: components_register: component hnp has no register or open function
[cn5:13991] mca: base: components_register: found loaded component pmi
[cn5:13991] mca: base: components_register: component pmi has no register or open function
[cn5:13991] mca: base: components_register: found loaded component singleton
[cn5:13991] mca: base: components_register: component singleton register function successful
[cn5:13991] mca: base: components_register: found loaded component slurm
[cn5:13991] mca: base: components_register: component slurm has no register or open function
[cn5:13991] mca: base: components_register: found loaded component tool
[cn5:13991] mca: base: components_register: component tool has no register or open function
[cn5:13991] mca: base: components_open: opening ess components
[cn5:13991] mca: base: components_open: found loaded component env
[cn5:13991] mca: base: components_open: component env open function successful
[cn5:13991] mca: base: components_open: found loaded component hnp
[cn5:13991] mca: base: components_open: component hnp open function successful
[cn5:13991] mca: base: components_open: found loaded component pmi
[cn5:13991] mca: base: components_open: component pmi open function successful
[cn5:13991] mca: base: components_open: found loaded component singleton
[cn5:13991] mca: base: components_open: component singleton open function successful
[cn5:13991] mca: base: components_open: found loaded component slurm
[cn5:13991] mca: base: components_open: component slurm open function successful
[cn5:13991] mca: base: components_open: found loaded component tool
[cn5:13991] mca: base: components_open: component tool open function successful
[cn5:13991] mca:base:select: Auto-selecting ess components
[cn5:13991] mca:base:select:(  ess) Querying component [env]
[cn5:13991] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn5:13991] mca:base:select:(  ess) Querying component [hnp]
[cn5:13991] mca:base:select:(  ess) Skipping component [hnp]. Query failed to return a module
[cn5:13991] mca:base:select:(  ess) Querying component [pmi]
[cn5:13991] mca:base:select:(  ess) Query of component [pmi] set priority to 35
[cn5:13991] mca:base:select:(  ess) Querying component [singleton]
[cn5:13991] mca:base:select:(  ess) Skipping component [singleton]. Query failed to return a module
[cn5:13991] mca:base:select:(  ess) Querying component [slurm]
[cn5:13991] mca:base:select:(  ess) Query of component [slurm] set priority to 50
[cn5:13991] mca:base:select:(  ess) Querying component [tool]
[cn5:13991] mca:base:select:(  ess) Skipping component [tool]. Query failed to return a module
[cn5:13991] mca:base:select:(  ess) Selected component [slurm]
[cn5:13991] mca: base: close: component env closed
[cn5:13991] mca: base: close: unloading component env
[cn5:13991] mca: base: close: component hnp closed
[cn5:13991] mca: base: close: unloading component hnp
[cn5:13991] mca: base: close: component pmi closed
[cn5:13991] mca: base: close: unloading component pmi
[cn5:13991] mca: base: close: component singleton closed
[cn5:13991] mca: base: close: unloading component singleton
[cn5:13991] mca: base: close: component tool closed
[cn5:13991] mca: base: close: unloading component tool
[cn2:00785] progress: initialized event flag to: 3
[cn2:00785] progress: initialized yield_when_idle to: false
[cn2:00785] progress: initialized num users to: 0
[cn2:00785] progress: initialized poll rate to: 23940000
[cn2:00785] progress: event_users_increment setting count to 1
[cn2:00785] mca: base: components_register: registering ess components
[cn2:00785] mca: base: components_register: found loaded component env
[cn2:00785] mca: base: components_register: component env has no register or open function
[cn2:00785] mca: base: components_register: found loaded component hnp
[cn2:00785] mca: base: components_register: component hnp has no register or open function
[cn2:00785] mca: base: components_register: found loaded component pmi
[cn2:00785] mca: base: components_register: component pmi has no register or open function
[cn2:00785] mca: base: components_register: found loaded component singleton
[cn2:00785] mca: base: components_register: component singleton register function successful
[cn2:00785] mca: base: components_register: found loaded component slurm
[cn2:00785] mca: base: components_register: component slurm has no register or open function
[cn2:00785] mca: base: components_register: found loaded component tool
[cn2:00785] mca: base: components_register: component tool has no register or open function
[cn2:00785] mca: base: components_open: opening ess components
[cn2:00785] mca: base: components_open: found loaded component env
[cn2:00785] mca: base: components_open: component env open function successful
[cn2:00785] mca: base: components_open: found loaded component hnp
[cn2:00785] mca: base: components_open: component hnp open function successful
[cn2:00785] mca: base: components_open: found loaded component pmi
[cn2:00785] mca: base: components_open: component pmi open function successful
[cn2:00785] mca: base: components_open: found loaded component singleton
[cn2:00785] mca: base: components_open: component singleton open function successful
[cn2:00785] mca: base: components_open: found loaded component slurm
[cn2:00785] mca: base: components_open: component slurm open function successful
[cn2:00785] mca: base: components_open: found loaded component tool
[cn2:00785] mca: base: components_open: component tool open function successful
[cn2:00785] mca:base:select: Auto-selecting ess components
[cn2:00785] mca:base:select:(  ess) Querying component [env]
[cn2:00785] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn2:00785] mca:base:select:(  ess) Querying component [hnp]
[cn2:00785] mca:base:select:(  ess) Skipping component [hnp]. Query failed to return a module
[cn2:00785] mca:base:select:(  ess) Querying component [pmi]
[cn5:13991] ess:slurm setting name
[cn5:13991] ess:slurm set name to [[15051,0],4]
[cn5:13991] ess:slurm set nodename to cn5
[cn4:17654] progress: initialized event flag to: 3
[cn4:17654] progress: initialized yield_when_idle to: false
[cn4:17654] progress: initialized num users to: 0
[cn4:17654] progress: initialized poll rate to: 23940000
[cn4:17654] progress: event_users_increment setting count to 1
[cn2:00785] mca:base:select:(  ess) Query of component [pmi] set priority to 35
[cn2:00785] mca:base:select:(  ess) Querying component [singleton]
[cn2:00785] mca:base:select:(  ess) Skipping component [singleton]. Query failed to return a module
[cn2:00785] mca:base:select:(  ess) Querying component [slurm]
[cn2:00785] mca:base:select:(  ess) Query of component [slurm] set priority to 50
[cn5:13991] [[15051,0],4] Topology Info:
[cn4:17654] mca: base: components_register: registering ess components
[cn4:17654] mca: base: components_register: found loaded component env
[cn4:17654] mca: base: components_register: component env has no register or open function
[cn4:17654] mca: base: components_register: found loaded component hnp
[cn4:17654] mca: base: components_register: component hnp has no register or open function
[cn4:17654] mca: base: components_register: found loaded component pmi
[cn4:17654] mca: base: components_register: component pmi has no register or open function
[cn2:00785] mca:base:select:(  ess) Querying component [tool]
[cn2:00785] mca:base:select:(  ess) Skipping component [tool]. Query failed to return a module
[cn2:00785] mca:base:select:(  ess) Selected component [slurm]
[cn2:00785] mca: base: close: component env closed
[cn2:00785] mca: base: close: unloading component env
[cn5:13991] Type: Machine Number of child objects: 2
	Name=NULL
	total=25100444KB
	DMIProductName=S5520UR
	DMIProductVersion=....................
	DMIBoardVendor="Intel Corporation"
	DMIBoardName=S5520UR
	DMIBoardVersion=E22554-751
	DMIBoardAssetTag=....................
	DMIChassisVendor=...............................
	DMIChassisType=23
	DMIChassisVersion=..................
	DMIChassisAssetTag=....................
	DMIBIOSVendor="Intel Corp."
	DMIBIOSVersion=S5500.86B.01.00.0054.092820101104
	DMIBIOSDate=09/28/2010
	DMISysVendor="Intel Corporation"
	Backend=Linux
	LinuxCgroup=/
	OSName=Linux
	OSRelease=2.6.32-431.3.1.el6.x86_64
	OSVersion="#1 SMP Fri Jan 3 21:39:27 UTC 2014"
	Architecture=x86_64
	Cpuset:  0x000000ff
	Online:  0x000000ff
	Allowed: 0x000000ff
	Bind CPU proc:   TRUE
	Bind CPU thread: TRUE
	Bind MEM proc:   FALSE
	Bind MEM thread: FALSE
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12517532KB
		total=12517532KB
		Cpuset:  0x00000055
		Online:  0x00000055
		Allowed: 0x00000055
[cn4:17654] mca: base: components_register: found loaded component singleton
[cn2:00785] mca: base: close: component hnp closed
[cn2:00785] mca: base: close: unloading component hnp
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x00000055
			Online:  0x00000055
			Allowed: 0x00000055
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x00000055
				Online:  0x00000055
				Allowed: 0x00000055
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000001
					Online:  0x00000001
					Allowed: 0x00000001
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000001
						Online:  0x00000001
						Allowed: 0x00000001
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000001
							Online:  0x00000001
							Allowed: 0x00000001
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000001
								Online:  0x00000001
								Allowed: 0x00000001
[cn4:17654] mca: base: components_register: component singleton register function successful
[cn4:17654] mca: base: components_register: found loaded component slurm
[cn4:17654] mca: base: components_register: component slurm has no register or open function
[cn2:00785] mca: base: close: component pmi closed
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000004
					Online:  0x00000004
					Allowed: 0x00000004
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000004
						Online:  0x00000004
						Allowed: 0x00000004
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000004
							Online:  0x00000004
							Allowed: 0x00000004
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000004
								Online:  0x00000004
								Allowed: 0x00000004
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000010
					Online:  0x00000010
					Allowed: 0x00000010
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000010
						Online:  0x00000010
						Allowed: 0x00000010
[cn4:17654] mca: base: components_register: found loaded component tool
[cn4:17654] mca: base: components_register: component tool has no register or open function
[cn2:00785] mca: base: close: unloading component pmi
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000010
							Online:  0x00000010
							Allowed: 0x00000010
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000010
								Online:  0x00000010
								Allowed: 0x00000010
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000040
					Online:  0x00000040
					Allowed: 0x00000040
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000040
						Online:  0x00000040
						Allowed: 0x00000040
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000040
							Online:  0x00000040
							Allowed: 0x00000040
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000040
								Online:  0x00000040
								Allowed: 0x00000040
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12582912KB
[cn4:17654] mca: base: components_open: opening ess components
[cn4:17654] mca: base: components_open: found loaded component env
[cn4:17654] mca: base: components_open: component env open function successful
[cn4:17654] mca: base: components_open: found loaded component hnp
[cn4:17654] mca: base: components_open: component hnp open function successful
[cn4:17654] mca: base: components_open: found loaded component pmi
[cn2:00785] mca: base: close: component singleton closed
[cn2:00785] mca: base: close: unloading component singleton
[cn2:00785] mca: base: close: component tool closed
[cn2:00785] mca: base: close: unloading component tool
		total=12582912KB
[cn4:17654] mca: base: components_open: component pmi open function successful
[cn4:17654] mca: base: components_open: found loaded component singleton
[cn4:17654] mca: base: components_open: component singleton open function successful
[cn4:17654] mca: base: components_open: found loaded component slurm
[cn4:17654] mca: base: components_open: component slurm open function successful
[cn4:17654] mca: base: components_open: found loaded component tool
[cn4:17654] mca: base: components_open: component tool open function successful
[cn4:17654] mca:base:select: Auto-selecting ess components
[cn4:17654] mca:base:select:(  ess) Querying component [env]
		Cpuset:  0x000000aa
		Online:  0x000000aa
		Allowed: 0x000000aa
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x000000aa
			Online:  0x000000aa
			Allowed: 0x000000aa
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x000000aa
				Online:  0x000000aa
				Allowed: 0x000000aa
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000002
					Online:  0x00000002
					Allowed: 0x00000002
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000002
						Online:  0x00000002
						Allowed: 0x00000002
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000002
							Online:  0x00000002
							Allowed: 0x00000002
							Type: PU Number of child objects: 0
								Name=NULL
[cn4:17654] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn4:17654] mca:base:select:(  ess) Querying component [hnp]
[cn4:17654] mca:base:select:(  ess) Skipping component [hnp]. Query failed to return a module
[cn4:17654] mca:base:select:(  ess) Querying component [pmi]
[cn2:00785] ess:slurm setting name
[cn2:00785] ess:slurm set name to [[15051,0],1]
[cn2:00785] ess:slurm set nodename to cn2
								Cpuset:  0x00000002
								Online:  0x00000002
								Allowed: 0x00000002
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000008
					Online:  0x00000008
					Allowed: 0x00000008
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000008
						Online:  0x00000008
						Allowed: 0x00000008
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000008
							Online:  0x00000008
							Allowed: 0x00000008
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000008
								Online:  0x00000008
								Allowed: 0x00000008
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000020
					Online:  0x00000020
					Allowed: 0x00000020
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
[cn4:17654] mca:base:select:(  ess) Query of component [pmi] set priority to 35
[cn4:17654] mca:base:select:(  ess) Querying component [singleton]
[cn4:17654] mca:base:select:(  ess) Skipping component [singleton]. Query failed to return a module
[cn4:17654] mca:base:select:(  ess) Querying component [slurm]
[cn4:17654] mca:base:select:(  ess) Query of component [slurm] set priority to 50
[cn4:17654] mca:base:select:(  ess) Querying component [tool]
[cn4:17654] mca:base:select:(  ess) Skipping component [tool]. Query failed to return a module
[cn4:17654] mca:base:select:(  ess) Selected component [slurm]
[cn4:17654] mca: base: close: component env closed
[cn4:17654] mca: base: close: unloading component env
[cn4:17654] mca: base: close: component hnp closed
[cn4:17654] mca: base: close: unloading component hnp
[cn4:17654] mca: base: close: component pmi closed
[cn4:17654] mca: base: close: unloading component pmi
[cn4:17654] mca: base: close: component singleton closed
[cn4:17654] mca: base: close: unloading component singleton
[cn4:17654] mca: base: close: component tool closed
[cn4:17654] mca: base: close: unloading component tool
[cn4:17654] ess:slurm setting name
[cn4:17654] ess:slurm set name to [[15051,0],3]
[cn4:17654] ess:slurm set nodename to cn4
						ways=8
						Cpuset:  0x00000020
						Online:  0x00000020
						Allowed: 0x00000020
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000020
							Online:  0x00000020
							Allowed: 0x00000020
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000020
								Online:  0x00000020
								Allowed: 0x00000020
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000080
					Online:  0x00000080
					Allowed: 0x00000080
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000080
						Online:  0x00000080
						Allowed: 0x00000080
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000080
							Online:  0x00000080
							Allowed: 0x00000080
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000080
								Online:  0x00000080
								Allowed: 0x00000080
[cn3:21242] progress: initialized event flag to: 3
[cn3:21242] progress: initialized yield_when_idle to: false
[cn3:21242] progress: initialized num users to: 0
[cn3:21242] progress: initialized poll rate to: 23940000
[cn3:21242] progress: event_users_increment setting count to 1
[cn3:21242] mca: base: components_register: registering ess components
[cn3:21242] mca: base: components_register: found loaded component env
[cn3:21242] mca: base: components_register: component env has no register or open function
[cn3:21242] mca: base: components_register: found loaded component hnp
[cn3:21242] mca: base: components_register: component hnp has no register or open function
[cn3:21242] mca: base: components_register: found loaded component pmi
[cn3:21242] mca: base: components_register: component pmi has no register or open function
[cn3:21242] mca: base: components_register: found loaded component singleton
[cn3:21242] mca: base: components_register: component singleton register function successful
[cn3:21242] mca: base: components_register: found loaded component slurm
[cn3:21242] mca: base: components_register: component slurm has no register or open function
[cn3:21242] mca: base: components_register: found loaded component tool
[cn3:21242] mca: base: components_register: component tool has no register or open function
[cn3:21242] mca: base: components_open: opening ess components
[cn3:21242] mca: base: components_open: found loaded component env
[cn3:21242] mca: base: components_open: component env open function successful
[cn3:21242] mca: base: components_open: found loaded component hnp
[cn3:21242] mca: base: components_open: component hnp open function successful
[cn3:21242] mca: base: components_open: found loaded component pmi
[cn3:21242] mca: base: components_open: component pmi open function successful
[cn3:21242] mca: base: components_open: found loaded component singleton
[cn3:21242] mca: base: components_open: component singleton open function successful
[cn3:21242] mca: base: components_open: found loaded component slurm
[cn3:21242] mca: base: components_open: component slurm open function successful
[cn3:21242] mca: base: components_open: found loaded component tool
[cn3:21242] mca: base: components_open: component tool open function successful
[cn3:21242] mca:base:select: Auto-selecting ess components
[cn3:21242] mca:base:select:(  ess) Querying component [env]
[cn3:21242] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn3:21242] mca:base:select:(  ess) Querying component [hnp]
[cn3:21242] mca:base:select:(  ess) Skipping component [hnp]. Query failed to return a module
[cn3:21242] mca:base:select:(  ess) Querying component [pmi]
[cn3:21242] mca:base:select:(  ess) Query of component [pmi] set priority to 35
[cn3:21242] mca:base:select:(  ess) Querying component [singleton]
[cn3:21242] mca:base:select:(  ess) Skipping component [singleton]. Query failed to return a module
[cn3:21242] mca:base:select:(  ess) Querying component [slurm]
[cn3:21242] mca:base:select:(  ess) Query of component [slurm] set priority to 50
[cn3:21242] mca:base:select:(  ess) Querying component [tool]
[cn3:21242] mca:base:select:(  ess) Skipping component [tool]. Query failed to return a module
[cn3:21242] mca:base:select:(  ess) Selected component [slurm]
[cn3:21242] mca: base: close: component env closed
[cn3:21242] mca: base: close: unloading component env
[cn3:21242] mca: base: close: component hnp closed
[cn3:21242] mca: base: close: unloading component hnp
[cn3:21242] mca: base: close: component pmi closed
[cn3:21242] mca: base: close: unloading component pmi
[cn3:21242] mca: base: close: component singleton closed
[cn3:21242] mca: base: close: unloading component singleton
[cn3:21242] mca: base: close: component tool closed
[cn3:21242] mca: base: close: unloading component tool
[cn3:21242] ess:slurm setting name
[cn4:17654] [[15051,0],3] Topology Info:
[cn2:00785] [[15051,0],1] Topology Info:
[cn4:17654] Type: Machine Number of child objects: 2
	Name=NULL
	total=25100516KB
	DMIProductName=S5520UR
	DMIProductVersion=....................
	DMIBoardVendor="Intel Corporation"
	DMIBoardName=S5520UR
	DMIBoardVersion=E22554-751
	DMIBoardAssetTag=....................
	DMIChassisVendor=...............................
	DMIChassisType=23
	DMIChassisVersion=..................
	DMIChassisAssetTag=....................
	DMIBIOSVendor="Intel Corp."
	DMIBIOSVersion=S5500.86B.01.00.0054.092820101104
	DMIBIOSDate=09/28/2010
	DMISysVendor="Intel Corporation"
	Backend=Linux
	LinuxCgroup=/
	OSName=Linux
	OSRelease=2.6.32-431.3.1.el6.x86_64
	OSVersion="#1 SMP Fri Jan 3 21:39:27 UTC 2014"
	Architecture=x86_64
	Cpuset:  0x000000ff
	Online:  0x000000ff
	Allowed: 0x000000ff
	Bind CPU proc:   TRUE
	Bind CPU thread: TRUE
	Bind MEM proc:   FALSE
	Bind MEM thread: FALSE
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12517604KB
		total=12517604KB
		Cpuset:  0x00000055
		Online:  0x00000055
		Allowed: 0x00000055
[cn2:00785] Type: Machine Number of child objects: 2
	Name=NULL
	total=25100516KB
	DMIProductName=S5520UR
	DMIProductVersion=....................
	DMIBoardVendor="Intel Corporation"
	DMIBoardName=S5520UR
	DMIBoardVersion=E22554-751
	DMIBoardAssetTag=....................
	DMIChassisVendor=...............................
	DMIChassisType=23
	DMIChassisVersion=..................
	DMIChassisAssetTag=....................
	DMIBIOSVendor="Intel Corp."
	DMIBIOSVersion=S5500.86B.01.00.0054.092820101104
	DMIBIOSDate=09/28/2010
	DMISysVendor="Intel Corporation"
	Backend=Linux
	LinuxCgroup=/
	OSName=Linux
	OSRelease=2.6.32-431.3.1.el6.x86_64
	OSVersion="#1 SMP Fri Jan 3 21:39:27 UTC 2014"
	Architecture=x86_64
	Cpuset:  0x000000ff
	Online:  0x000000ff
	Allowed: 0x000000ff
	Bind CPU proc:   TRUE
	Bind CPU thread: TRUE
	Bind MEM proc:   FALSE
	Bind MEM thread: FALSE
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12517604KB
		total=12517604KB
		Cpuset:  0x00000055
		Online:  0x00000055
		Allowed: 0x00000055
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x00000055
			Online:  0x00000055
			Allowed: 0x00000055
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x00000055
				Online:  0x00000055
				Allowed: 0x00000055
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000001
					Online:  0x00000001
					Allowed: 0x00000001
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000001
						Online:  0x00000001
						Allowed: 0x00000001
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000001
							Online:  0x00000001
							Allowed: 0x00000001
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000001
								Online:  0x00000001
								Allowed: 0x00000001
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x00000055
			Online:  0x00000055
			Allowed: 0x00000055
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x00000055
				Online:  0x00000055
				Allowed: 0x00000055
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000001
					Online:  0x00000001
					Allowed: 0x00000001
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000001
						Online:  0x00000001
						Allowed: 0x00000001
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000001
							Online:  0x00000001
							Allowed: 0x00000001
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000001
								Online:  0x00000001
								Allowed: 0x00000001
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000004
					Online:  0x00000004
					Allowed: 0x00000004
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000004
						Online:  0x00000004
						Allowed: 0x00000004
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000004
							Online:  0x00000004
							Allowed: 0x00000004
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000004
								Online:  0x00000004
								Allowed: 0x00000004
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000010
					Online:  0x00000010
					Allowed: 0x00000010
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000010
						Online:  0x00000010
						Allowed: 0x00000010
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000004
					Online:  0x00000004
					Allowed: 0x00000004
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000004
						Online:  0x00000004
						Allowed: 0x00000004
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000004
							Online:  0x00000004
							Allowed: 0x00000004
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000004
								Online:  0x00000004
								Allowed: 0x00000004
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000010
					Online:  0x00000010
					Allowed: 0x00000010
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000010
						Online:  0x00000010
						Allowed: 0x00000010
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000010
							Online:  0x00000010
							Allowed: 0x00000010
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000010
								Online:  0x00000010
								Allowed: 0x00000010
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000040
					Online:  0x00000040
					Allowed: 0x00000040
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000040
						Online:  0x00000040
						Allowed: 0x00000040
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000040
							Online:  0x00000040
							Allowed: 0x00000040
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000040
								Online:  0x00000040
								Allowed: 0x00000040
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12582912KB
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000010
							Online:  0x00000010
							Allowed: 0x00000010
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000010
								Online:  0x00000010
								Allowed: 0x00000010
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000040
					Online:  0x00000040
					Allowed: 0x00000040
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000040
						Online:  0x00000040
						Allowed: 0x00000040
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000040
							Online:  0x00000040
							Allowed: 0x00000040
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000040
								Online:  0x00000040
								Allowed: 0x00000040
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12582912KB
		total=12582912KB
		total=12582912KB
		Cpuset:  0x000000aa
		Online:  0x000000aa
		Allowed: 0x000000aa
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x000000aa
			Online:  0x000000aa
			Allowed: 0x000000aa
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x000000aa
				Online:  0x000000aa
				Allowed: 0x000000aa
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000002
					Online:  0x00000002
					Allowed: 0x00000002
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000002
						Online:  0x00000002
						Allowed: 0x00000002
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000002
							Online:  0x00000002
							Allowed: 0x00000002
							Type: PU Number of child objects: 0
								Name=NULL
		Cpuset:  0x000000aa
		Online:  0x000000aa
		Allowed: 0x000000aa
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x000000aa
			Online:  0x000000aa
			Allowed: 0x000000aa
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x000000aa
				Online:  0x000000aa
				Allowed: 0x000000aa
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000002
					Online:  0x00000002
					Allowed: 0x00000002
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000002
						Online:  0x00000002
						Allowed: 0x00000002
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000002
							Online:  0x00000002
							Allowed: 0x00000002
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000002
								Online:  0x00000002
								Allowed: 0x00000002
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000008
					Online:  0x00000008
					Allowed: 0x00000008
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000008
						Online:  0x00000008
						Allowed: 0x00000008
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000008
							Online:  0x00000008
							Allowed: 0x00000008
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000008
								Online:  0x00000008
								Allowed: 0x00000008
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000020
					Online:  0x00000020
					Allowed: 0x00000020
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
								Cpuset:  0x00000002
								Online:  0x00000002
								Allowed: 0x00000002
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000008
					Online:  0x00000008
					Allowed: 0x00000008
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000008
						Online:  0x00000008
						Allowed: 0x00000008
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000008
							Online:  0x00000008
							Allowed: 0x00000008
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000008
								Online:  0x00000008
								Allowed: 0x00000008
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000020
					Online:  0x00000020
					Allowed: 0x00000020
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000020
						Online:  0x00000020
						Allowed: 0x00000020
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000020
							Online:  0x00000020
							Allowed: 0x00000020
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000020
								Online:  0x00000020
								Allowed: 0x00000020
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000080
					Online:  0x00000080
					Allowed: 0x00000080
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000080
						Online:  0x00000080
						Allowed: 0x00000080
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000080
							Online:  0x00000080
							Allowed: 0x00000080
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000080
								Online:  0x00000080
						ways=8
						Cpuset:  0x00000020
						Online:  0x00000020
						Allowed: 0x00000020
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000020
							Online:  0x00000020
							Allowed: 0x00000020
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000020
								Online:  0x00000020
								Allowed: 0x00000020
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000080
					Online:  0x00000080
					Allowed: 0x00000080
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000080
						Online:  0x00000080
						Allowed: 0x00000080
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000080
							Online:  0x00000080
							Allowed: 0x00000080
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000080
								Online:  0x00000080
								Allowed: 0x00000080
								Allowed: 0x00000080
[cn5:13991] mca: base: components_register: registering oob components
[cn5:13991] mca: base: components_register: found loaded component tcp
[cn5:13991] mca: base: components_register: component tcp register function successful
[cn5:13991] mca: base: components_open: opening oob components
[cn5:13991] mca: base: components_open: found loaded component tcp
[cn5:13991] mca: base: components_open: component tcp open function successful
[cn5:13991] mca:oob:select: checking available component tcp
[cn5:13991] mca:oob:select: Querying component [tcp]
[cn5:13991] oob:tcp: component_available called
[cn5:13991] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn5:13991] [[15051,0],4] oob:tcp:init rejecting loopback interface lo
[cn5:13991] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn5:13991] [[15051,0],4] oob:tcp:init creating module for V4 address on interface eth0
[cn5:13991] [[15051,0],4] creating OOB-TCP module for interface eth0
[cn5:13991] [[15051,0],4] oob:tcp:init adding 192.168.1.5 to our list of V4 connections
[cn5:13991] [[15051,0],4] TCP STARTUP
[cn5:13991] [[15051,0],4] attempting to bind to IPv4 port 0
[cn5:13991] [[15051,0],4] assigned IPv4 port 46667
[cn5:13991] mca:oob:select: Adding component to end
[cn5:13991] mca:oob:select: Found 1 active transports
[cn5:13991] mca: base: components_register: registering rml components
[cn5:13991] mca: base: components_register: found loaded component oob
[cn5:13991] mca: base: components_register: component oob has no register or open function
[cn5:13991] mca: base: components_open: opening rml components
[cn5:13991] mca: base: components_open: found loaded component oob
[cn5:13991] mca: base: components_open: component oob open function successful
[cn5:13991] orte_rml_base_select: initializing rml component oob
[cn3:21242] ess:slurm set name to [[15051,0],2]
[cn3:21242] ess:slurm set nodename to cn3
[cn3:21242] [[15051,0],2] Topology Info:
[cn3:21242] Type: Machine Number of child objects: 2
	Name=NULL
	total=25100444KB
	DMIProductName=S5520UR
	DMIProductVersion=....................
	DMIBoardVendor="Intel Corporation"
	DMIBoardName=S5520UR
	DMIBoardVersion=E22554-751
	DMIBoardAssetTag=....................
	DMIChassisVendor=...............................
	DMIChassisType=23
	DMIChassisVersion=..................
	DMIChassisAssetTag=....................
	DMIBIOSVendor="Intel Corp."
	DMIBIOSVersion=S5500.86B.01.00.0054.092820101104
	DMIBIOSDate=09/28/2010
	DMISysVendor="Intel Corporation"
	Backend=Linux
	LinuxCgroup=/
	OSName=Linux
	OSRelease=2.6.32-431.3.1.el6.x86_64
	OSVersion="#1 SMP Fri Jan 3 21:39:27 UTC 2014"
	Architecture=x86_64
	Cpuset:  0x000000ff
	Online:  0x000000ff
	Allowed: 0x000000ff
	Bind CPU proc:   TRUE
	Bind CPU thread: TRUE
	Bind MEM proc:   FALSE
	Bind MEM thread: FALSE
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12517532KB
		total=12517532KB
		Cpuset:  0x00000055
		Online:  0x00000055
		Allowed: 0x00000055
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x00000055
			Online:  0x00000055
			Allowed: 0x00000055
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x00000055
				Online:  0x00000055
				Allowed: 0x00000055
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000001
					Online:  0x00000001
					Allowed: 0x00000001
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000001
						Online:  0x00000001
						Allowed: 0x00000001
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000001
							Online:  0x00000001
							Allowed: 0x00000001
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000001
								Online:  0x00000001
								Allowed: 0x00000001
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000004
					Online:  0x00000004
					Allowed: 0x00000004
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000004
						Online:  0x00000004
						Allowed: 0x00000004
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000004
							Online:  0x00000004
							Allowed: 0x00000004
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000004
								Online:  0x00000004
								Allowed: 0x00000004
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000010
					Online:  0x00000010
					Allowed: 0x00000010
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000010
						Online:  0x00000010
						Allowed: 0x00000010
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000010
							Online:  0x00000010
							Allowed: 0x00000010
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000010
								Online:  0x00000010
								Allowed: 0x00000010
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000040
					Online:  0x00000040
					Allowed: 0x00000040
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000040
						Online:  0x00000040
						Allowed: 0x00000040
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000040
							Online:  0x00000040
							Allowed: 0x00000040
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000040
								Online:  0x00000040
								Allowed: 0x00000040
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12582912KB
		total=12582912KB
		Cpuset:  0x000000aa
		Online:  0x000000aa
		Allowed: 0x000000aa
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x000000aa
			Online:  0x000000aa
			Allowed: 0x000000aa
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x000000aa
				Online:  0x000000aa
				Allowed: 0x000000aa
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000002
					Online:  0x00000002
					Allowed: 0x00000002
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000002
						Online:  0x00000002
						Allowed: 0x00000002
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000002
							Online:  0x00000002
							Allowed: 0x00000002
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000002
								Online:  0x00000002
								Allowed: 0x00000002
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000008
					Online:  0x00000008
					Allowed: 0x00000008
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000008
						Online:  0x00000008
						Allowed: 0x00000008
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000008
							Online:  0x00000008
							Allowed: 0x00000008
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000008
								Online:  0x00000008
								Allowed: 0x00000008
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000020
					Online:  0x00000020
					Allowed: 0x00000020
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000020
						Online:  0x00000020
						Allowed: 0x00000020
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000020
							Online:  0x00000020
							Allowed: 0x00000020
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000020
								Online:  0x00000020
								Allowed: 0x00000020
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000080
					Online:  0x00000080
					Allowed: 0x00000080
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000080
						Online:  0x00000080
						Allowed: 0x00000080
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000080
							Online:  0x00000080
							Allowed: 0x00000080
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000080
								Online:  0x00000080
								Allowed: 0x00000080
[cn5:13991] mca: base: components_register: registering routed components
[cn5:13991] mca: base: components_register: found loaded component binomial
[cn5:13991] mca: base: components_register: component binomial has no register or open function
[cn5:13991] mca: base: components_register: found loaded component debruijn
[cn5:13991] mca: base: components_register: component debruijn has no register or open function
[cn5:13991] mca: base: components_register: found loaded component direct
[cn5:13991] mca: base: components_register: component direct has no register or open function
[cn5:13991] mca: base: components_register: found loaded component radix
[cn5:13991] mca: base: components_register: component radix register function successful
[cn5:13991] mca: base: components_open: opening routed components
[cn5:13991] mca: base: components_open: found loaded component binomial
[cn5:13991] mca: base: components_open: found loaded component debruijn
[cn5:13991] mca: base: components_open: found loaded component direct
[cn5:13991] mca: base: components_open: found loaded component radix
[cn5:13991] mca:base:select: Auto-selecting routed components
[cn5:13991] mca:base:select:(routed) Querying component [binomial]
[cn5:13991] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn5:13991] mca:base:select:(routed) Querying component [debruijn]
[cn5:13991] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn5:13991] mca:base:select:(routed) Querying component [direct]
[cn5:13991] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn5:13991] mca:base:select:(routed) Querying component [radix]
[cn5:13991] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn5:13991] mca:base:select:(routed) Selected component [binomial]
[cn5:13991] mca: base: close: unloading component debruijn
[cn5:13991] mca: base: close: unloading component direct
[cn5:13991] mca: base: close: unloading component radix
[cn5:13991] orte_routed_base_select: initializing selected component binomial
[cn5:13991] mca: base: components_register: registering dstore components
[cn5:13991] mca: base: components_register: found loaded component hash
[cn5:13991] mca: base: components_register: component hash register function successful
[cn5:13991] mca: base: components_open: opening dstore components
[cn5:13991] mca: base: components_open: found loaded component hash
[cn5:13991] mca:dstore:select: checking available component hash
[cn5:13991] mca:dstore:select: Querying component [hash]
[cn5:13991] mca: base: components_register: registering grpcomm components
[cn5:13991] mca: base: components_register: found loaded component bad
[cn5:13991] mca: base: components_register: component bad has no register or open function
[cn5:13991] mca: base: components_register: found loaded component pmi
[cn5:13991] mca: base: components_register: component pmi register function successful
[cn5:13991] mca: base: components_open: opening grpcomm components
[cn5:13991] mca: base: components_open: found loaded component bad
[cn5:13991] mca: base: components_open: component bad open function successful
[cn5:13991] mca: base: components_open: found loaded component pmi
[cn5:13991] mca: base: components_open: component pmi open function successful
[cn5:13991] mca:base:select: Auto-selecting grpcomm components
[cn5:13991] mca:base:select:(grpcomm) Querying component [bad]
[cn5:13991] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn5:13991] mca:base:select:(grpcomm) Querying component [pmi]
[cn5:13991] mca:base:select:(grpcomm) Skipping component [pmi]. Query failed to return a module
[cn5:13991] mca:base:select:(grpcomm) Selected component [bad]
[cn5:13991] mca: base: close: component pmi closed
[cn5:13991] mca: base: close: unloading component pmi
[cn5:13991] [[15051,0],4] grpcomm:base:receive start comm
[cn5:13991] [[15051,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn5:13991] [[15051,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 15
[cn5:13991] [[15051,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 32
[cn5:13991] [[15051,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 11
[cn5:13991] [[15051,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn5:13991] [[15051,0],4] routed:binomial rank 0 parent 0 me 4 num_procs 5
[cn5:13991] [[15051,0],4] routed:binomial find children of rank 0
[cn5:13991] [[15051,0],4] routed:binomial find children checking peer 1
[cn5:13991] [[15051,0],4] routed:binomial find children computing tree
[cn5:13991] [[15051,0],4] routed:binomial rank 1 parent 0 me 4 num_procs 5
[cn5:13991] [[15051,0],4] routed:binomial find children of rank 1
[cn5:13991] [[15051,0],4] routed:binomial find children checking peer 3
[cn5:13991] [[15051,0],4] routed:binomial find children computing tree
[cn5:13991] [[15051,0],4] routed:binomial rank 3 parent 1 me 4 num_procs 5
[cn5:13991] [[15051,0],4] routed:binomial find children of rank 3
[cn5:13991] [[15051,0],4] routed:binomial find children checking peer 7
[cn5:13991] [[15051,0],4] routed:binomial find children checking peer 5
[cn5:13991] [[15051,0],4] routed:binomial find children checking peer 2
[cn5:13991] [[15051,0],4] routed:binomial find children computing tree
[cn5:13991] [[15051,0],4] routed:binomial rank 2 parent 0 me 4 num_procs 5
[cn5:13991] [[15051,0],4] routed:binomial find children of rank 2
[cn5:13991] [[15051,0],4] routed:binomial find children checking peer 6
[cn5:13991] [[15051,0],4] routed:binomial find children checking peer 4
[cn5:13991] [[15051,0],4] routed:binomial find children computing tree
[cn5:13991] [[15051,0],4] routed:binomial rank 4 parent 0 me 4 num_procs 5
[cn5:13991] [[15051,0],4] routed:binomial find children returning found value 0
[cn5:13991] [[15051,0],4]: parent 0 num_children 0
[cn5:13991] [[15051,0],4] setting up session dir with
	tmpdir: UNDEF
	host cn5
[cn2:00785] mca: base: components_register: registering oob components
[cn2:00785] mca: base: components_register: found loaded component tcp
[cn4:17654] mca: base: components_register: registering oob components
[cn4:17654] mca: base: components_register: found loaded component tcp
[cn2:00785] mca: base: components_register: component tcp register function successful
[cn4:17654] mca: base: components_register: component tcp register function successful
[cn2:00785] mca: base: components_open: opening oob components
[cn2:00785] mca: base: components_open: found loaded component tcp
[cn2:00785] mca: base: components_open: component tcp open function successful
[cn2:00785] mca:oob:select: checking available component tcp
[cn2:00785] mca:oob:select: Querying component [tcp]
[cn2:00785] oob:tcp: component_available called
[cn4:17654] mca: base: components_open: opening oob components
[cn4:17654] mca: base: components_open: found loaded component tcp
[cn4:17654] mca: base: components_open: component tcp open function successful
[cn4:17654] mca:oob:select: checking available component tcp
[cn4:17654] mca:oob:select: Querying component [tcp]
[cn4:17654] oob:tcp: component_available called
[cn2:00785] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn2:00785] [[15051,0],1] oob:tcp:init rejecting loopback interface lo
[cn2:00785] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn2:00785] [[15051,0],1] oob:tcp:init creating module for V4 address on interface eth0
[cn2:00785] [[15051,0],1] creating OOB-TCP module for interface eth0
[cn2:00785] [[15051,0],1] oob:tcp:init adding 192.168.1.2 to our list of V4 connections
[cn2:00785] [[15051,0],1] TCP STARTUP
[cn4:17654] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn4:17654] [[15051,0],3] oob:tcp:init rejecting loopback interface lo
[cn4:17654] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn4:17654] [[15051,0],3] oob:tcp:init creating module for V4 address on interface eth0
[cn4:17654] [[15051,0],3] creating OOB-TCP module for interface eth0
[cn4:17654] [[15051,0],3] oob:tcp:init adding 192.168.1.4 to our list of V4 connections
[cn4:17654] [[15051,0],3] TCP STARTUP
[cn4:17654] [[15051,0],3] attempting to bind to IPv4 port 0
[cn2:00785] [[15051,0],1] attempting to bind to IPv4 port 0
[cn4:17654] [[15051,0],3] assigned IPv4 port 54546
[cn2:00785] [[15051,0],1] assigned IPv4 port 41722
[cn2:00785] mca:oob:select: Adding component to end
[cn4:17654] mca:oob:select: Adding component to end
[cn4:17654] mca:oob:select: Found 1 active transports
[cn2:00785] mca:oob:select: Found 1 active transports
[cn2:00785] mca: base: components_register: registering rml components
[cn2:00785] mca: base: components_register: found loaded component oob
[cn2:00785] mca: base: components_register: component oob has no register or open function
[cn4:17654] mca: base: components_register: registering rml components
[cn4:17654] mca: base: components_register: found loaded component oob
[cn4:17654] mca: base: components_register: component oob has no register or open function
[cn4:17654] mca: base: components_open: opening rml components
[cn4:17654] mca: base: components_open: found loaded component oob
[cn4:17654] mca: base: components_open: component oob open function successful
[cn4:17654] orte_rml_base_select: initializing rml component oob
[cn2:00785] mca: base: components_open: opening rml components
[cn2:00785] mca: base: components_open: found loaded component oob
[cn2:00785] mca: base: components_open: component oob open function successful
[cn2:00785] orte_rml_base_select: initializing rml component oob
[cn2:00785] mca: base: components_register: registering routed components
[cn2:00785] mca: base: components_register: found loaded component binomial
[cn2:00785] mca: base: components_register: component binomial has no register or open function
[cn2:00785] mca: base: components_register: found loaded component debruijn
[cn2:00785] mca: base: components_register: component debruijn has no register or open function
[cn2:00785] mca: base: components_register: found loaded component direct
[cn2:00785] mca: base: components_register: component direct has no register or open function
[cn2:00785] mca: base: components_register: found loaded component radix
[cn2:00785] mca: base: components_register: component radix register function successful
[cn2:00785] mca: base: components_open: opening routed components
[cn2:00785] mca: base: components_open: found loaded component binomial
[cn2:00785] mca: base: components_open: found loaded component debruijn
[cn2:00785] mca: base: components_open: found loaded component direct
[cn2:00785] mca: base: components_open: found loaded component radix
[cn2:00785] mca:base:select: Auto-selecting routed components
[cn2:00785] mca:base:select:(routed) Querying component [binomial]
[cn2:00785] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn2:00785] mca:base:select:(routed) Querying component [debruijn]
[cn2:00785] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn2:00785] mca:base:select:(routed) Querying component [direct]
[cn2:00785] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn2:00785] mca:base:select:(routed) Querying component [radix]
[cn2:00785] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn2:00785] mca:base:select:(routed) Selected component [binomial]
[cn2:00785] mca: base: close: unloading component debruijn
[cn2:00785] mca: base: close: unloading component direct
[cn2:00785] mca: base: close: unloading component radix
[cn2:00785] orte_routed_base_select: initializing selected component binomial
[cn4:17654] mca: base: components_register: registering routed components
[cn4:17654] mca: base: components_register: found loaded component binomial
[cn4:17654] mca: base: components_register: component binomial has no register or open function
[cn4:17654] mca: base: components_register: found loaded component debruijn
[cn4:17654] mca: base: components_register: component debruijn has no register or open function
[cn4:17654] mca: base: components_register: found loaded component direct
[cn4:17654] mca: base: components_register: component direct has no register or open function
[cn4:17654] mca: base: components_register: found loaded component radix
[cn4:17654] mca: base: components_register: component radix register function successful
[cn4:17654] mca: base: components_open: opening routed components
[cn4:17654] mca: base: components_open: found loaded component binomial
[cn4:17654] mca: base: components_open: found loaded component debruijn
[cn4:17654] mca: base: components_open: found loaded component direct
[cn4:17654] mca: base: components_open: found loaded component radix
[cn4:17654] mca:base:select: Auto-selecting routed components
[cn4:17654] mca:base:select:(routed) Querying component [binomial]
[cn4:17654] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn4:17654] mca:base:select:(routed) Querying component [debruijn]
[cn4:17654] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn4:17654] mca:base:select:(routed) Querying component [direct]
[cn4:17654] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn4:17654] mca:base:select:(routed) Querying component [radix]
[cn4:17654] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn4:17654] mca:base:select:(routed) Selected component [binomial]
[cn4:17654] mca: base: close: unloading component debruijn
[cn4:17654] mca: base: close: unloading component direct
[cn4:17654] mca: base: close: unloading component radix
[cn4:17654] orte_routed_base_select: initializing selected component binomial
[cn3:21242] mca: base: components_register: registering oob components
[cn3:21242] mca: base: components_register: found loaded component tcp
[cn3:21242] mca: base: components_register: component tcp register function successful
[cn3:21242] mca: base: components_open: opening oob components
[cn3:21242] mca: base: components_open: found loaded component tcp
[cn3:21242] mca: base: components_open: component tcp open function successful
[cn3:21242] mca:oob:select: checking available component tcp
[cn3:21242] mca:oob:select: Querying component [tcp]
[cn3:21242] oob:tcp: component_available called
[cn3:21242] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn3:21242] [[15051,0],2] oob:tcp:init rejecting loopback interface lo
[cn3:21242] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn3:21242] [[15051,0],2] oob:tcp:init creating module for V4 address on interface eth0
[cn3:21242] [[15051,0],2] creating OOB-TCP module for interface eth0
[cn3:21242] [[15051,0],2] oob:tcp:init adding 192.168.1.3 to our list of V4 connections
[cn3:21242] [[15051,0],2] TCP STARTUP
[cn3:21242] [[15051,0],2] attempting to bind to IPv4 port 0
[cn3:21242] [[15051,0],2] assigned IPv4 port 49500
[cn3:21242] mca:oob:select: Adding component to end
[cn3:21242] mca:oob:select: Found 1 active transports
[cn5:13991] procdir: /tmp/openmpi-sessions-artpol@cn5_0/15051/0/4
[cn5:13991] jobdir: /tmp/openmpi-sessions-artpol@cn5_0/15051/0
[cn5:13991] top: openmpi-sessions-artpol@cn5_0
[cn5:13991] tmp: /tmp
[cn3:21242] mca: base: components_register: registering rml components
[cn3:21242] mca: base: components_register: found loaded component oob
[cn3:21242] mca: base: components_register: component oob has no register or open function
[cn3:21242] mca: base: components_open: opening rml components
[cn3:21242] mca: base: components_open: found loaded component oob
[cn3:21242] mca: base: components_open: component oob open function successful
[cn3:21242] orte_rml_base_select: initializing rml component oob
[cn5:13991] sess_dir_cleanup: job session dir does not exist
[cn5:13991] procdir: /tmp/openmpi-sessions-artpol@cn5_0/15051/0/4
[cn5:13991] jobdir: /tmp/openmpi-sessions-artpol@cn5_0/15051/0
[cn5:13991] top: openmpi-sessions-artpol@cn5_0
[cn5:13991] tmp: /tmp
[cn5:13991] [[15051,0],4] routed_binomial: init routes for daemon job [15051,0]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn5:13991] [[15051,0],4] routed_binomial: completed init routes
[cn5:13991] [[15051,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 3
[cn5:13991] [[15051,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 21
[cn5:13991] [[15051,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 45
[cn5:13991] [[15051,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 46
[cn5:13991] [[15051,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 1
Daemon [[15051,0],4] checking in as pid 13991 on host cn5
[cn5:13991] progress: progress_set_yield_when_idle to false
[cn5:13991] progress: set_event_flag setting to 1
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 10
[cn5:13991] [[15051,0],4] orted: up and running - waiting for commands!
[cn5:13991] [[15051,0],4] posting recv
[cn5:13991] [[15051,0],4] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn5:13991] [[15051,0],4] posting recv
[cn5:13991] [[15051,0],4] posting persistent recv on tag 15 for peer [[WILDCARD],WILDCARD]
[cn5:13991] [[15051,0],4] posting recv
[cn5:13991] [[15051,0],4] posting persistent recv on tag 32 for peer [[WILDCARD],WILDCARD]
[cn5:13991] [[15051,0],4] posting recv
[cn5:13991] [[15051,0],4] posting persistent recv on tag 11 for peer [[WILDCARD],WILDCARD]
[cn5:13991] [[15051,0],4] posting recv
[cn5:13991] [[15051,0],4] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn5:13991] [[15051,0],4]: set_addr to uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn5:13991] [[15051,0],4]:set_addr checking if peer [[15051,0],0] is reachable via component tcp
[cn5:13991] [[15051,0],4] oob:tcp: working peer [[15051,0],0] address tcp://192.168.1.1,192.168.10.2:37478
[cn5:13991] [[15051,0],4] PEER [[15051,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:13991] [[15051,0],4] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn5:13991] [[15051,0],4]:tcp set addr for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn5:13991] [[15051,0],4]: peer [[15051,0],0] is reachable via component tcp
[cn5:13991] [[15051,0],4] posting recv
[cn5:13991] [[15051,0],4] posting persistent recv on tag 3 for peer [[WILDCARD],WILDCARD]
[cn5:13991] [[15051,0],4] posting recv
[cn5:13991] [[15051,0],4] posting persistent recv on tag 21 for peer [[WILDCARD],WILDCARD]
[cn5:13991] [[15051,0],4] posting recv
[cn5:13991] [[15051,0],4] posting persistent recv on tag 45 for peer [[WILDCARD],WILDCARD]
[cn5:13991] [[15051,0],4] posting recv
[cn5:13991] [[15051,0],4] posting persistent recv on tag 46 for peer [[WILDCARD],WILDCARD]
[cn5:13991] [[15051,0],4] posting recv
[cn5:13991] [[15051,0],4] posting persistent recv on tag 1 for peer [[WILDCARD],WILDCARD]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 10
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4]:tcp:processing set_peer cmd for interface eth0
[cn5:13991] [[15051,0],4] SET_PEER ADDING PEER [[15051,0],0]
[cn5:13991] [[15051,0],4] set_peer: peer [[15051,0],0] is listening on interface eth0 net 192.168.1.1 port 37478
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:10
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:10
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: initiating connection to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],0]
[cn5:13991] [[15051,0],4] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] via interface eth0
[cn5:13991] [[15051,0],4] oob:tcp:peer creating socket to [[15051,0],0]
[cn5:13991] [[15051,0],4] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] via interface eth0 on socket 9
[cn5:13991] [[15051,0],4] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] on 192.168.1.1:37478 - 0 retries
[cn1:13986] [[15051,0],0] mca_oob_tcp_listen_thread: new connection: (14, 0) 192.168.1.5:52020
[cn5:13991] [[15051,0],4] waiting for connect completion to [[15051,0],0] - activating send event
[cn1:13986] [[15051,0],0] connection_handler: working connection (14, 2) 192.168.1.5:52020
[cn1:13986] [[15051,0],0] accept_connection: 192.168.1.5:52020
[cn1:13986] [[15051,0],0]:tcp:recv:handler called
[cn1:13986] [[15051,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 14
[cn1:13986] [[15051,0],0] waiting for connect ack from UNKNOWN
[cn1:13986] [[15051,0],0] connect ack received from UNKNOWN
[cn1:13986] [[15051,0],0] connect-ack recvd from UNKNOWN
[cn1:13986] [[15051,0],0] mca_oob_tcp_recv_connect: connection from new peer
[cn1:13986] [[15051,0],0] connect-ack header from [[15051,0],4] is okay
[cn1:13986] [[15051,0],0] waiting for connect ack from [[15051,0],4]
[cn1:13986] [[15051,0],0] connect ack received from [[15051,0],4]
[cn1:13986] [[15051,0],0] connect-ack version from [[15051,0],4] matches ours
[cn1:13986] [[15051,0],0] connect-ack [[15051,0],4] authenticated
[cn1:13986] [[15051,0],0] tcp:peer_accept called for peer [[15051,0],4] in state UNKNOWN on socket 14
[cn1:13986] [[15051,0],0] SEND CONNECT ACK
[cn1:13986] fetching data from INTERNAL dstore
[cn1:13986] dstore:hash:fetch: searching for key opal.cred on proc 986382336
[cn1:13986] dstore_hash:fetch key opal.cred for proc 986382336 not found
[cn1:13986] [[15051,0],0] send blocking of 47 bytes to socket 14
[cn1:13986] [[15051,0],0] blocking send complete to socket 14
[cn1:13986] [[15051,0],0]-[[15051,0],4] tcp_peer_connected on socket 14
[cn1:13986] [[15051,0],0] routed_binomial_update: [[15051,0],4] --> [[15051,0],4]
[cn1:13986] [[15051,0],0]-[[15051,0],4] accepted: 192.168.1.1 - 192.168.1.5 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:13986] [[15051,0],0] tcp:set_module called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler CONNECTING
[cn5:13991] [[15051,0],4]:tcp:complete_connect called for peer [[15051,0],0] on socket 9
[cn5:13991] [[15051,0],4] tcp_peer_complete_connect: sending ack to [[15051,0],0]
[cn5:13991] [[15051,0],4] SEND CONNECT ACK
[cn5:13991] fetching data from INTERNAL dstore
[cn5:13991] dstore:hash:fetch: searching for key opal.cred on proc 18166251520
[cn5:13991] dstore_hash:fetch key opal.cred for proc 18166251520 not found
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] send blocking of 47 bytes to socket 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] blocking send complete to socket 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] tcp_peer_complete_connect: setting read event on connection to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] RECV CONNECT ACK FROM [[15051,0],0] ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 96
[cn5:13991] [[15051,0],4] waiting for connect ack from [[15051,0],0]
[cn5:13991] [[15051,0],4] connect ack received from [[15051,0],0]
[cn5:13991] [[15051,0],4] connect-ack recvd from [[15051,0],0]
[cn5:13991] [[15051,0],4] connect-ack header from [[15051,0],0] is okay
[cn5:13991] [[15051,0],4] waiting for connect ack from [[15051,0],0]
[cn5:13991] [[15051,0],4] connect ack received from [[15051,0],0]
[cn5:13991] [[15051,0],4] connect-ack version from [[15051,0],0] matches ours
[cn5:13991] [[15051,0],4] connect-ack [[15051,0],0] authenticated
[cn5:13991] [[15051,0],4]-[[15051,0],0] tcp_peer_connected on socket 9
[cn5:13991] [[15051,0],4] routed_binomial_update: [[15051,0],0] --> [[15051,0],0]
[cn5:13991] [[15051,0],4]-[[15051,0],0] connected: 192.168.1.5 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn5:13991] [[15051,0],4]:tcp:recv:handler starting send/recv events
[cn5:13991] [[15051,0],4] tcp:set_module called for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 96 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] mca: base: components_register: registering dstore components
[cn2:00785] mca: base: components_register: found loaded component hash
[cn2:00785] mca: base: components_register: component hash register function successful
[cn2:00785] mca: base: components_open: opening dstore components
[cn2:00785] mca: base: components_open: found loaded component hash
[cn2:00785] mca:dstore:select: checking available component hash
[cn2:00785] mca:dstore:select: Querying component [hash]
[cn2:00785] mca: base: components_register: registering grpcomm components
[cn2:00785] mca: base: components_register: found loaded component bad
[cn2:00785] mca: base: components_register: component bad has no register or open function
[cn2:00785] mca: base: components_register: found loaded component pmi
[cn2:00785] mca: base: components_register: component pmi register function successful
[cn2:00785] mca: base: components_open: opening grpcomm components
[cn2:00785] mca: base: components_open: found loaded component bad
[cn2:00785] mca: base: components_open: component bad open function successful
[cn2:00785] mca: base: components_open: found loaded component pmi
[cn2:00785] mca: base: components_open: component pmi open function successful
[cn2:00785] mca:base:select: Auto-selecting grpcomm components
[cn2:00785] mca:base:select:(grpcomm) Querying component [bad]
[cn2:00785] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn2:00785] mca:base:select:(grpcomm) Querying component [pmi]
[cn2:00785] mca:base:select:(grpcomm) Skipping component [pmi]. Query failed to return a module
[cn2:00785] mca:base:select:(grpcomm) Selected component [bad]
[cn2:00785] mca: base: close: component pmi closed
[cn2:00785] mca: base: close: unloading component pmi
[cn2:00785] [[15051,0],1] grpcomm:base:receive start comm
[cn2:00785] [[15051,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn2:00785] [[15051,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 15
[cn2:00785] [[15051,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 32
[cn2:00785] [[15051,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 11
[cn4:17654] mca: base: components_register: registering dstore components
[cn4:17654] mca: base: components_register: found loaded component hash
[cn4:17654] mca: base: components_register: component hash register function successful
[cn4:17654] mca: base: components_open: opening dstore components
[cn4:17654] mca: base: components_open: found loaded component hash
[cn4:17654] mca:dstore:select: checking available component hash
[cn4:17654] mca:dstore:select: Querying component [hash]
[cn4:17654] mca: base: components_register: registering grpcomm components
[cn4:17654] mca: base: components_register: found loaded component bad
[cn4:17654] mca: base: components_register: component bad has no register or open function
[cn4:17654] mca: base: components_register: found loaded component pmi
[cn4:17654] mca: base: components_register: component pmi register function successful
[cn4:17654] mca: base: components_open: opening grpcomm components
[cn4:17654] mca: base: components_open: found loaded component bad
[cn4:17654] mca: base: components_open: component bad open function successful
[cn4:17654] mca: base: components_open: found loaded component pmi
[cn4:17654] mca: base: components_open: component pmi open function successful
[cn4:17654] mca:base:select: Auto-selecting grpcomm components
[cn4:17654] mca:base:select:(grpcomm) Querying component [bad]
[cn4:17654] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn4:17654] mca:base:select:(grpcomm) Querying component [pmi]
[cn4:17654] mca:base:select:(grpcomm) Skipping component [pmi]. Query failed to return a module
[cn4:17654] mca:base:select:(grpcomm) Selected component [bad]
[cn4:17654] mca: base: close: component pmi closed
[cn4:17654] mca: base: close: unloading component pmi
[cn4:17654] [[15051,0],3] grpcomm:base:receive start comm
[cn4:17654] [[15051,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn4:17654] [[15051,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 15
[cn4:17654] [[15051,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 32
[cn4:17654] [[15051,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 11
[cn3:21242] mca: base: components_register: registering routed components
[cn3:21242] mca: base: components_register: found loaded component binomial
[cn3:21242] mca: base: components_register: component binomial has no register or open function
[cn3:21242] mca: base: components_register: found loaded component debruijn
[cn3:21242] mca: base: components_register: component debruijn has no register or open function
[cn3:21242] mca: base: components_register: found loaded component direct
[cn3:21242] mca: base: components_register: component direct has no register or open function
[cn3:21242] mca: base: components_register: found loaded component radix
[cn3:21242] mca: base: components_register: component radix register function successful
[cn3:21242] mca: base: components_open: opening routed components
[cn3:21242] mca: base: components_open: found loaded component binomial
[cn3:21242] mca: base: components_open: found loaded component debruijn
[cn3:21242] mca: base: components_open: found loaded component direct
[cn3:21242] mca: base: components_open: found loaded component radix
[cn3:21242] mca:base:select: Auto-selecting routed components
[cn3:21242] mca:base:select:(routed) Querying component [binomial]
[cn3:21242] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn3:21242] mca:base:select:(routed) Querying component [debruijn]
[cn3:21242] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn3:21242] mca:base:select:(routed) Querying component [direct]
[cn3:21242] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn3:21242] mca:base:select:(routed) Querying component [radix]
[cn3:21242] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn3:21242] mca:base:select:(routed) Selected component [binomial]
[cn3:21242] mca: base: close: unloading component debruijn
[cn3:21242] mca: base: close: unloading component direct
[cn3:21242] mca: base: close: unloading component radix
[cn3:21242] orte_routed_base_select: initializing selected component binomial
[cn2:00785] [[15051,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn2:00785] [[15051,0],1] routed:binomial rank 0 parent 0 me 1 num_procs 5
[cn2:00785] [[15051,0],1] routed:binomial find children of rank 0
[cn2:00785] [[15051,0],1] routed:binomial find children checking peer 1
[cn2:00785] [[15051,0],1] routed:binomial find children computing tree
[cn2:00785] [[15051,0],1] routed:binomial rank 1 parent 0 me 1 num_procs 5
[cn2:00785] [[15051,0],1] routed:binomial 1 found child 3
[cn2:00785] [[15051,0],1] routed:binomial rank 0 parent 0 me 3 num_procs 5
[cn2:00785] [[15051,0],1] routed:binomial find children of rank 0
[cn2:00785] [[15051,0],1] routed:binomial find children checking peer 1
[cn2:00785] [[15051,0],1] routed:binomial find children computing tree
[cn2:00785] [[15051,0],1] routed:binomial rank 1 parent 0 me 3 num_procs 5
[cn2:00785] [[15051,0],1] routed:binomial find children of rank 1
[cn2:00785] [[15051,0],1] routed:binomial find children checking peer 3
[cn2:00785] [[15051,0],1] routed:binomial find children computing tree
[cn2:00785] [[15051,0],1] routed:binomial rank 3 parent 1 me 3 num_procs 5
[cn2:00785] [[15051,0],1] routed:binomial find children returning found value 1
[cn2:00785] [[15051,0],1] routed:binomial find children returning found value 1
[cn2:00785] [[15051,0],1] routed:binomial find children returning found value 0
[cn2:00785] [[15051,0],1]: parent 0 num_children 1
[cn2:00785] [[15051,0],1]: 	child 3
[cn2:00785] [[15051,0],1] setting up session dir with
	tmpdir: UNDEF
	host cn2
[cn4:17654] [[15051,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn4:17654] [[15051,0],3] routed:binomial rank 0 parent 0 me 3 num_procs 5
[cn4:17654] [[15051,0],3] routed:binomial find children of rank 0
[cn4:17654] [[15051,0],3] routed:binomial find children checking peer 1
[cn4:17654] [[15051,0],3] routed:binomial find children computing tree
[cn4:17654] [[15051,0],3] routed:binomial rank 1 parent 0 me 3 num_procs 5
[cn4:17654] [[15051,0],3] routed:binomial find children of rank 1
[cn4:17654] [[15051,0],3] routed:binomial find children checking peer 3
[cn4:17654] [[15051,0],3] routed:binomial find children computing tree
[cn4:17654] [[15051,0],3] routed:binomial rank 3 parent 1 me 3 num_procs 5
[cn4:17654] [[15051,0],3] routed:binomial find children returning found value 1
[cn4:17654] [[15051,0],3] routed:binomial find children returning found value 1
[cn4:17654] [[15051,0],3]: parent 1 num_children 0
[cn4:17654] [[15051,0],3] setting up session dir with
	tmpdir: UNDEF
	host cn4
[cn4:17654] procdir: /tmp/openmpi-sessions-artpol@cn4_0/15051/0/3
[cn4:17654] jobdir: /tmp/openmpi-sessions-artpol@cn4_0/15051/0
[cn4:17654] top: openmpi-sessions-artpol@cn4_0
[cn4:17654] tmp: /tmp
[cn4:17654] sess_dir_cleanup: job session dir does not exist
[cn4:17654] procdir: /tmp/openmpi-sessions-artpol@cn4_0/15051/0/3
[cn4:17654] jobdir: /tmp/openmpi-sessions-artpol@cn4_0/15051/0
[cn4:17654] top: openmpi-sessions-artpol@cn4_0
[cn4:17654] tmp: /tmp
[cn4:17654] [[15051,0],3] routed_binomial: init routes for daemon job [15051,0]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn4:17654] [[15051,0],3] routed_binomial: completed init routes
[cn3:21242] mca: base: components_register: registering dstore components
[cn3:21242] mca: base: components_register: found loaded component hash
[cn3:21242] mca: base: components_register: component hash register function successful
[cn3:21242] mca: base: components_open: opening dstore components
[cn3:21242] mca: base: components_open: found loaded component hash
[cn3:21242] mca:dstore:select: checking available component hash
[cn3:21242] mca:dstore:select: Querying component [hash]
[cn3:21242] mca: base: components_register: registering grpcomm components
[cn3:21242] mca: base: components_register: found loaded component bad
[cn3:21242] mca: base: components_register: component bad has no register or open function
[cn3:21242] mca: base: components_register: found loaded component pmi
[cn3:21242] mca: base: components_register: component pmi register function successful
[cn3:21242] mca: base: components_open: opening grpcomm components
[cn3:21242] mca: base: components_open: found loaded component bad
[cn3:21242] mca: base: components_open: component bad open function successful
[cn3:21242] mca: base: components_open: found loaded component pmi
[cn3:21242] mca: base: components_open: component pmi open function successful
[cn3:21242] mca:base:select: Auto-selecting grpcomm components
[cn3:21242] mca:base:select:(grpcomm) Querying component [bad]
[cn3:21242] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn3:21242] mca:base:select:(grpcomm) Querying component [pmi]
[cn3:21242] mca:base:select:(grpcomm) Skipping component [pmi]. Query failed to return a module
[cn3:21242] mca:base:select:(grpcomm) Selected component [bad]
[cn3:21242] mca: base: close: component pmi closed
[cn3:21242] mca: base: close: unloading component pmi
[cn3:21242] [[15051,0],2] grpcomm:base:receive start comm
[cn3:21242] [[15051,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn3:21242] [[15051,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 15
[cn3:21242] [[15051,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 32
[cn3:21242] [[15051,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 11
[cn3:21242] [[15051,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn3:21242] [[15051,0],2] routed:binomial rank 0 parent 0 me 2 num_procs 5
[cn3:21242] [[15051,0],2] routed:binomial find children of rank 0
[cn3:21242] [[15051,0],2] routed:binomial find children checking peer 1
[cn3:21242] [[15051,0],2] routed:binomial find children computing tree
[cn3:21242] [[15051,0],2] routed:binomial rank 1 parent 0 me 2 num_procs 5
[cn3:21242] [[15051,0],2] routed:binomial find children of rank 1
[cn3:21242] [[15051,0],2] routed:binomial find children checking peer 3
[cn3:21242] [[15051,0],2] routed:binomial find children computing tree
[cn3:21242] [[15051,0],2] routed:binomial rank 3 parent 1 me 2 num_procs 5
[cn3:21242] [[15051,0],2] routed:binomial find children of rank 3
[cn3:21242] [[15051,0],2] routed:binomial find children checking peer 7
[cn3:21242] [[15051,0],2] routed:binomial find children checking peer 5
[cn3:21242] [[15051,0],2] routed:binomial find children checking peer 2
[cn3:21242] [[15051,0],2] routed:binomial find children computing tree
[cn3:21242] [[15051,0],2] routed:binomial rank 2 parent 0 me 2 num_procs 5
[cn3:21242] [[15051,0],2] routed:binomial find children returning found value 0
[cn3:21242] [[15051,0],2]: parent 0 num_children 0
[cn3:21242] [[15051,0],2] setting up session dir with
	tmpdir: UNDEF
	host cn3
[cn3:21242] procdir: /tmp/openmpi-sessions-artpol@cn3_0/15051/0/2
[cn3:21242] jobdir: /tmp/openmpi-sessions-artpol@cn3_0/15051/0
[cn3:21242] top: openmpi-sessions-artpol@cn3_0
[cn3:21242] tmp: /tmp
[cn3:21242] sess_dir_cleanup: job session dir does not exist
[cn2:00785] procdir: /tmp/openmpi-sessions-artpol@cn2_0/15051/0/1
[cn2:00785] jobdir: /tmp/openmpi-sessions-artpol@cn2_0/15051/0
[cn2:00785] top: openmpi-sessions-artpol@cn2_0
[cn2:00785] tmp: /tmp
[cn2:00785] sess_dir_cleanup: job session dir does not exist
[cn2:00785] procdir: /tmp/openmpi-sessions-artpol@cn2_0/15051/0/1
[cn2:00785] jobdir: /tmp/openmpi-sessions-artpol@cn2_0/15051/0
[cn2:00785] top: openmpi-sessions-artpol@cn2_0
[cn2:00785] tmp: /tmp
[cn2:00785] [[15051,0],1] routed_binomial: init routes for daemon job [15051,0]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn2:00785] [[15051,0],1] routed_binomial: completed init routes
[cn2:00785] [[15051,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 3
[cn2:00785] [[15051,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 21
[cn2:00785] [[15051,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 45
[cn2:00785] [[15051,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 46
[cn2:00785] [[15051,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 1
Daemon [[15051,0],1] checking in as pid 785 on host cn2
[cn2:00785] progress: progress_set_yield_when_idle to false
[cn2:00785] progress: set_event_flag setting to 1
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 10
[cn2:00785] [[15051,0],1] orted: up and running - waiting for commands!
[cn2:00785] [[15051,0],1] posting recv
[cn2:00785] [[15051,0],1] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn2:00785] [[15051,0],1] posting recv
[cn2:00785] [[15051,0],1] posting persistent recv on tag 15 for peer [[WILDCARD],WILDCARD]
[cn2:00785] [[15051,0],1] posting recv
[cn2:00785] [[15051,0],1] posting persistent recv on tag 32 for peer [[WILDCARD],WILDCARD]
[cn2:00785] [[15051,0],1] posting recv
[cn2:00785] [[15051,0],1] posting persistent recv on tag 11 for peer [[WILDCARD],WILDCARD]
[cn2:00785] [[15051,0],1] posting recv
[cn2:00785] [[15051,0],1] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn2:00785] [[15051,0],1]: set_addr to uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn2:00785] [[15051,0],1]:set_addr checking if peer [[15051,0],0] is reachable via component tcp
[cn2:00785] [[15051,0],1] oob:tcp: working peer [[15051,0],0] address tcp://192.168.1.1,192.168.10.2:37478
[cn2:00785] [[15051,0],1] PEER [[15051,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn2:00785] [[15051,0],1] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00785] [[15051,0],1]:tcp set addr for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn2:00785] [[15051,0],1]: peer [[15051,0],0] is reachable via component tcp
[cn2:00785] [[15051,0],1] posting recv
[cn2:00785] [[15051,0],1] posting persistent recv on tag 3 for peer [[WILDCARD],WILDCARD]
[cn2:00785] [[15051,0],1] posting recv
[cn2:00785] [[15051,0],1] posting persistent recv on tag 21 for peer [[WILDCARD],WILDCARD]
[cn2:00785] [[15051,0],1] posting recv
[cn2:00785] [[15051,0],1] posting persistent recv on tag 45 for peer [[WILDCARD],WILDCARD]
[cn2:00785] [[15051,0],1] posting recv
[cn2:00785] [[15051,0],1] posting persistent recv on tag 46 for peer [[WILDCARD],WILDCARD]
[cn2:00785] [[15051,0],1] posting recv
[cn2:00785] [[15051,0],1] posting persistent recv on tag 1 for peer [[WILDCARD],WILDCARD]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 10
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1]:tcp:processing set_peer cmd for interface eth0
[cn2:00785] [[15051,0],1] SET_PEER ADDING PEER [[15051,0],0]
[cn2:00785] [[15051,0],1] set_peer: peer [[15051,0],0] is listening on interface eth0 net 192.168.1.1 port 37478
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:10
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] mca_oob_tcp_listen_thread: new connection: (15, 11) 192.168.1.2:51057
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:10
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: initiating connection to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],0]
[cn2:00785] [[15051,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] via interface eth0
[cn2:00785] [[15051,0],1] oob:tcp:peer creating socket to [[15051,0],0]
[cn2:00785] [[15051,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] via interface eth0 on socket 9
[cn2:00785] [[15051,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] on 192.168.1.1:37478 - 0 retries
[cn2:00785] [[15051,0],1] waiting for connect completion to [[15051,0],0] - activating send event
[cn1:13986] [[15051,0],0] connection_handler: working connection (15, 11) 192.168.1.2:51057
[cn1:13986] [[15051,0],0] accept_connection: 192.168.1.2:51057
[cn1:13986] [[15051,0],0]:tcp:recv:handler called
[cn1:13986] [[15051,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 15
[cn1:13986] [[15051,0],0] waiting for connect ack from UNKNOWN
[cn1:13986] [[15051,0],0] connect ack received from UNKNOWN
[cn1:13986] [[15051,0],0] connect-ack recvd from UNKNOWN
[cn1:13986] [[15051,0],0] mca_oob_tcp_recv_connect: connection from new peer
[cn1:13986] [[15051,0],0] connect-ack header from [[15051,0],1] is okay
[cn1:13986] [[15051,0],0] waiting for connect ack from [[15051,0],1]
[cn1:13986] [[15051,0],0] connect ack received from [[15051,0],1]
[cn1:13986] [[15051,0],0] connect-ack version from [[15051,0],1] matches ours
[cn1:13986] [[15051,0],0] connect-ack [[15051,0],1] authenticated
[cn1:13986] [[15051,0],0] tcp:peer_accept called for peer [[15051,0],1] in state UNKNOWN on socket 15
[cn1:13986] [[15051,0],0] SEND CONNECT ACK
[cn1:13986] [[15051,0],0] send blocking of 47 bytes to socket 15
[cn1:13986] [[15051,0],0] blocking send complete to socket 15
[cn1:13986] [[15051,0],0]-[[15051,0],1] tcp_peer_connected on socket 15
[cn1:13986] [[15051,0],0] routed_binomial_update: [[15051,0],1] --> [[15051,0],1]
[cn1:13986] [[15051,0],0]-[[15051,0],1] accepted: 192.168.1.1 - 192.168.1.2 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:13986] [[15051,0],0] tcp:set_module called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler CONNECTING
[cn2:00785] [[15051,0],1]:tcp:complete_connect called for peer [[15051,0],0] on socket 9
[cn2:00785] [[15051,0],1] tcp_peer_complete_connect: sending ack to [[15051,0],0]
[cn2:00785] [[15051,0],1] SEND CONNECT ACK
[cn2:00785] fetching data from INTERNAL dstore
[cn2:00785] dstore:hash:fetch: searching for key opal.cred on proc 5281349632
[cn2:00785] dstore_hash:fetch key opal.cred for proc 5281349632 not found
[cn2:00785] [[15051,0],1] send blocking of 47 bytes to socket 9
[cn2:00785] [[15051,0],1] blocking send complete to socket 9
[cn2:00785] [[15051,0],1] tcp_peer_complete_connect: setting read event on connection to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 13097
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] RECV CONNECT ACK FROM [[15051,0],0] ON SOCKET 9
[cn2:00785] [[15051,0],1] waiting for connect ack from [[15051,0],0]
[cn2:00785] [[15051,0],1] connect ack received from [[15051,0],0]
[cn2:00785] [[15051,0],1] connect-ack recvd from [[15051,0],0]
[cn2:00785] [[15051,0],1] connect-ack header from [[15051,0],0] is okay
[cn2:00785] [[15051,0],1] waiting for connect ack from [[15051,0],0]
[cn2:00785] [[15051,0],1] connect ack received from [[15051,0],0]
[cn2:00785] [[15051,0],1] connect-ack version from [[15051,0],0] matches ours
[cn2:00785] [[15051,0],1] connect-ack [[15051,0],0] authenticated
[cn2:00785] [[15051,0],1]-[[15051,0],0] tcp_peer_connected on socket 9
[cn2:00785] [[15051,0],1] routed_binomial_update: [[15051,0],0] --> [[15051,0],0]
[cn2:00785] [[15051,0],1]-[[15051,0],0] connected: 192.168.1.2 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn2:00785] [[15051,0],1]:tcp:recv:handler starting send/recv events
[cn2:00785] [[15051,0],1] tcp:set_module called for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 13097 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 13097 BYTES FOR DEST [[15051,0],0] TAG 10
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 13097 bytes from [[15051,0],1] for tag 10
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch from daemon [[15051,0],1]
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch from daemon [[15051,0],1] on node cn2
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch attempting to assign daemon [[15051,0],1] to node cn2
[cn4:17654] [[15051,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 3
[cn4:17654] [[15051,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 21
[cn4:17654] [[15051,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 45
[cn4:17654] [[15051,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 46
[cn4:17654] [[15051,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 1
Daemon [[15051,0],3] checking in as pid 17654 on host cn4
[cn4:17654] progress: progress_set_yield_when_idle to false
[cn4:17654] progress: set_event_flag setting to 1
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 10
[cn4:17654] [[15051,0],3] orted: up and running - waiting for commands!
[cn4:17654] [[15051,0],3] posting recv
[cn4:17654] [[15051,0],3] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn4:17654] [[15051,0],3] posting recv
[cn4:17654] [[15051,0],3] posting persistent recv on tag 15 for peer [[WILDCARD],WILDCARD]
[cn4:17654] [[15051,0],3] posting recv
[cn4:17654] [[15051,0],3] posting persistent recv on tag 32 for peer [[WILDCARD],WILDCARD]
[cn4:17654] [[15051,0],3] posting recv
[cn4:17654] [[15051,0],3] posting persistent recv on tag 11 for peer [[WILDCARD],WILDCARD]
[cn4:17654] [[15051,0],3] posting recv
[cn4:17654] [[15051,0],3] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn4:17654] [[15051,0],3]: set_addr to uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn4:17654] [[15051,0],3]:set_addr checking if peer [[15051,0],0] is reachable via component tcp
[cn4:17654] [[15051,0],3] oob:tcp: working peer [[15051,0],0] address tcp://192.168.1.1,192.168.10.2:37478
[cn4:17654] [[15051,0],3] PEER [[15051,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17654] [[15051,0],3] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn4:17654] [[15051,0],3]:tcp set addr for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn4:17654] [[15051,0],3]: peer [[15051,0],0] is reachable via component tcp
[cn4:17654] [[15051,0],3] posting recv
[cn4:17654] [[15051,0],3] posting persistent recv on tag 3 for peer [[WILDCARD],WILDCARD]
[cn4:17654] [[15051,0],3] posting recv
[cn4:17654] [[15051,0],3] posting persistent recv on tag 21 for peer [[WILDCARD],WILDCARD]
[cn4:17654] [[15051,0],3] posting recv
[cn4:17654] [[15051,0],3] posting persistent recv on tag 45 for peer [[WILDCARD],WILDCARD]
[cn4:17654] [[15051,0],3] posting recv
[cn4:17654] [[15051,0],3] posting persistent recv on tag 46 for peer [[WILDCARD],WILDCARD]
[cn4:17654] [[15051,0],3] posting recv
[cn4:17654] [[15051,0],3] posting persistent recv on tag 1 for peer [[WILDCARD],WILDCARD]
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 10
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3]:tcp:processing set_peer cmd for interface eth0
[cn4:17654] [[15051,0],3] SET_PEER ADDING PEER [[15051,0],0]
[cn4:17654] [[15051,0],3] set_peer: peer [[15051,0],0] is listening on interface eth0 net 192.168.1.1 port 37478
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] RECEIVED TOPOLOGY FROM NODE cn2
[cn1:13986] [[15051,0],0] mca_oob_tcp_listen_thread: new connection: (16, 11) 192.168.1.4:60949
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:10
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:10
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: initiating connection to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],0]
[cn4:17654] [[15051,0],3] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] via interface eth0
[cn4:17654] [[15051,0],3] oob:tcp:peer creating socket to [[15051,0],0]
[cn4:17654] [[15051,0],3] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] via interface eth0 on socket 9
[cn4:17654] [[15051,0],3] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] on 192.168.1.1:37478 - 0 retries
[cn4:17654] [[15051,0],3] waiting for connect completion to [[15051,0],0] - activating send event
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler CONNECTING
[cn4:17654] [[15051,0],3]:tcp:complete_connect called for peer [[15051,0],0] on socket 9
[cn4:17654] [[15051,0],3] tcp_peer_complete_connect: sending ack to [[15051,0],0]
[cn4:17654] [[15051,0],3] SEND CONNECT ACK
[cn4:17654] fetching data from INTERNAL dstore
[cn4:17654] dstore:hash:fetch: searching for key opal.cred on proc 13871284224
[cn4:17654] dstore_hash:fetch key opal.cred for proc 13871284224 not found
[cn4:17654] [[15051,0],3] send blocking of 47 bytes to socket 9
[cn4:17654] [[15051,0],3] blocking send complete to socket 9
[cn4:17654] [[15051,0],3] tcp_peer_complete_connect: setting read event on connection to [[15051,0],0]
[cn1:13986] Type: Machine Number of child objects: 2
	Name=NULL
	total=25100516KB
	DMIProductName=S5520UR
	DMIProductVersion=....................
	DMIBoardVendor="Intel Corporation"
	DMIBoardName=S5520UR
	DMIBoardVersion=E22554-751
	DMIBoardAssetTag=....................
	DMIChassisVendor=...............................
	DMIChassisType=23
	DMIChassisVersion=..................
	DMIChassisAssetTag=....................
	DMIBIOSVendor="Intel Corp."
	DMIBIOSVersion=S5500.86B.01.00.0054.092820101104
	DMIBIOSDate=09/28/2010
	DMISysVendor="Intel Corporation"
	Backend=Linux
	LinuxCgroup=/
	OSName=Linux
	OSRelease=2.6.32-431.3.1.el6.x86_64
	OSVersion="#1 SMP Fri Jan 3 21:39:27 UTC 2014"
	Architecture=x86_64
	Cpuset:  0x000000ff
	Online:  0x000000ff
	Allowed: 0x000000ff
	Bind CPU proc:   TRUE
	Bind CPU thread: TRUE
	Bind MEM proc:   FALSE
	Bind MEM thread: FALSE
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12517604KB
		total=12517604KB
		Cpuset:  0x00000055
		Online:  0x00000055
		Allowed: 0x00000055
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x00000055
			Online:  0x00000055
			Allowed: 0x00000055
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x00000055
				Online:  0x00000055
				Allowed: 0x00000055
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000001
					Online:  0x00000001
					Allowed: 0x00000001
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000001
						Online:  0x00000001
						Allowed: 0x00000001
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000001
							Online:  0x00000001
							Allowed: 0x00000001
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000001
								Online:  0x00000001
								Allowed: 0x00000001
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000004
					Online:  0x00000004
					Allowed: 0x00000004
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000004
						Online:  0x00000004
						Allowed: 0x00000004
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000004
							Online:  0x00000004
							Allowed: 0x00000004
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000004
								Online:  0x00000004
								Allowed: 0x00000004
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000010
					Online:  0x00000010
					Allowed: 0x00000010
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000010
						Online:  0x00000010
						Allowed: 0x00000010
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000010
							Online:  0x00000010
							Allowed: 0x00000010
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000010
								Online:  0x00000010
								Allowed: 0x00000010
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000040
					Online:  0x00000040
					Allowed: 0x00000040
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000040
						Online:  0x00000040
						Allowed: 0x00000040
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000040
							Online:  0x00000040
							Allowed: 0x00000040
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000040
								Online:  0x00000040
								Allowed: 0x00000040
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12582912KB
		total=12582912KB
		Cpuset:  0x000000aa
		Online:  0x000000aa
		Allowed: 0x000000aa
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x000000aa
			Online:  0x000000aa
			Allowed: 0x000000aa
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x000000aa
				Online:  0x000000aa
				Allowed: 0x000000aa
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000002
					Online:  0x00000002
					Allowed: 0x00000002
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000002
						Online:  0x00000002
						Allowed: 0x00000002
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000002
							Online:  0x00000002
							Allowed: 0x00000002
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000002
								Online:  0x00000002
								Allowed: 0x00000002
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000008
					Online:  0x00000008
					Allowed: 0x00000008
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000008
						Online:  0x00000008
						Allowed: 0x00000008
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000008
							Online:  0x00000008
							Allowed: 0x00000008
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000008
								Online:  0x00000008
								Allowed: 0x00000008
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000020
					Online:  0x00000020
					Allowed: 0x00000020
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000020
						Online:  0x00000020
						Allowed: 0x00000020
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000020
							Online:  0x00000020
							Allowed: 0x00000020
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000020
								Online:  0x00000020
								Allowed: 0x00000020
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000080
					Online:  0x00000080
					Allowed: 0x00000080
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000080
						Online:  0x00000080
						Allowed: 0x00000080
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000080
							Online:  0x00000080
							Allowed: 0x00000080
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000080
								Online:  0x00000080
								Allowed: 0x00000080
[cn1:13986] [[15051,0],0] TOPOLOGY MATCHES - DISCARDING
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch completed for daemon [[15051,0],1] at contact 986382336.1;tcp://192.168.1.2:41722
[cn1:13986] [[15051,0],0]: set_addr to uri 986382336.1;tcp://192.168.1.2:41722
[cn1:13986] [[15051,0],0]:set_addr checking if peer [[15051,0],1] is reachable via component tcp
[cn1:13986] [[15051,0],0] oob:tcp: working peer [[15051,0],1] address tcp://192.168.1.2:41722
[cn1:13986] [[15051,0],0] PEER [[15051,0],1] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:13986] [[15051,0],0] PASSING ADDR 192.168.1.2 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:13986] [[15051,0],0]:tcp set addr for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]: peer [[15051,0],1] is reachable via component tcp
[cn1:13986] [[15051,0],0] connection_handler: working connection (16, 0) 192.168.1.4:60949
[cn1:13986] [[15051,0],0] accept_connection: 192.168.1.4:60949
[cn1:13986] [[15051,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:13986] [[15051,0],0] set_peer: peer [[15051,0],1] is listening on interface eth0 net 192.168.1.2 port 41722
[cn1:13986] [[15051,0],0]:tcp:recv:handler called
[cn1:13986] [[15051,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 16
[cn1:13986] [[15051,0],0] waiting for connect ack from UNKNOWN
[cn1:13986] [[15051,0],0] connect ack received from UNKNOWN
[cn1:13986] [[15051,0],0] connect-ack recvd from UNKNOWN
[cn1:13986] [[15051,0],0] mca_oob_tcp_recv_connect: connection from new peer
[cn1:13986] [[15051,0],0] connect-ack header from [[15051,0],3] is okay
[cn1:13986] [[15051,0],0] waiting for connect ack from [[15051,0],3]
[cn1:13986] [[15051,0],0] connect ack received from [[15051,0],3]
[cn1:13986] [[15051,0],0] connect-ack version from [[15051,0],3] matches ours
[cn1:13986] [[15051,0],0] connect-ack [[15051,0],3] authenticated
[cn1:13986] [[15051,0],0] tcp:peer_accept called for peer [[15051,0],3] in state UNKNOWN on socket 16
[cn1:13986] [[15051,0],0] SEND CONNECT ACK
[cn1:13986] [[15051,0],0] send blocking of 47 bytes to socket 16
[cn1:13986] [[15051,0],0] blocking send complete to socket 16
[cn1:13986] [[15051,0],0]-[[15051,0],3] tcp_peer_connected on socket 16
[cn1:13986] [[15051,0],0] routed_binomial_update: [[15051,0],3] --> [[15051,0],3]
[cn1:13986] [[15051,0],0]-[[15051,0],3] accepted: 192.168.1.1 - 192.168.1.4 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:13986] [[15051,0],0] tcp:set_module called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 96
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] RECV CONNECT ACK FROM [[15051,0],0] ON SOCKET 9
[cn4:17654] [[15051,0],3] waiting for connect ack from [[15051,0],0]
[cn4:17654] [[15051,0],3] connect ack received from [[15051,0],0]
[cn4:17654] [[15051,0],3] connect-ack recvd from [[15051,0],0]
[cn4:17654] [[15051,0],3] connect-ack header from [[15051,0],0] is okay
[cn4:17654] [[15051,0],3] waiting for connect ack from [[15051,0],0]
[cn4:17654] [[15051,0],3] connect ack received from [[15051,0],0]
[cn4:17654] [[15051,0],3] connect-ack version from [[15051,0],0] matches ours
[cn4:17654] [[15051,0],3] connect-ack [[15051,0],0] authenticated
[cn4:17654] [[15051,0],3]-[[15051,0],0] tcp_peer_connected on socket 9
[cn4:17654] [[15051,0],3] routed_binomial_update: [[15051,0],0] --> [[15051,0],0]
[cn4:17654] [[15051,0],3]-[[15051,0],0] connected: 192.168.1.4 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn4:17654] [[15051,0],3]:tcp:recv:handler starting send/recv events
[cn4:17654] [[15051,0],3] tcp:set_module called for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 96 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] procdir: /tmp/openmpi-sessions-artpol@cn3_0/15051/0/2
[cn3:21242] jobdir: /tmp/openmpi-sessions-artpol@cn3_0/15051/0
[cn3:21242] top: openmpi-sessions-artpol@cn3_0
[cn3:21242] tmp: /tmp
[cn3:21242] [[15051,0],2] routed_binomial: init routes for daemon job [15051,0]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn3:21242] [[15051,0],2] routed_binomial: completed init routes
[cn3:21242] [[15051,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 3
[cn3:21242] [[15051,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 21
[cn3:21242] [[15051,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 45
[cn3:21242] [[15051,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 46
[cn3:21242] [[15051,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 1
Daemon [[15051,0],2] checking in as pid 21242 on host cn3
[cn3:21242] progress: progress_set_yield_when_idle to false
[cn3:21242] progress: set_event_flag setting to 1
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 10
[cn3:21242] [[15051,0],2] orted: up and running - waiting for commands!
[cn3:21242] [[15051,0],2] posting recv
[cn3:21242] [[15051,0],2] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn3:21242] [[15051,0],2] posting recv
[cn3:21242] [[15051,0],2] posting persistent recv on tag 15 for peer [[WILDCARD],WILDCARD]
[cn3:21242] [[15051,0],2] posting recv
[cn3:21242] [[15051,0],2] posting persistent recv on tag 32 for peer [[WILDCARD],WILDCARD]
[cn3:21242] [[15051,0],2] posting recv
[cn3:21242] [[15051,0],2] posting persistent recv on tag 11 for peer [[WILDCARD],WILDCARD]
[cn3:21242] [[15051,0],2] posting recv
[cn3:21242] [[15051,0],2] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn3:21242] [[15051,0],2]: set_addr to uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn3:21242] [[15051,0],2]:set_addr checking if peer [[15051,0],0] is reachable via component tcp
[cn3:21242] [[15051,0],2] oob:tcp: working peer [[15051,0],0] address tcp://192.168.1.1,192.168.10.2:37478
[cn3:21242] [[15051,0],2] PEER [[15051,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn3:21242] [[15051,0],2] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21242] [[15051,0],2]:tcp set addr for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn3:21242] [[15051,0],2]: peer [[15051,0],0] is reachable via component tcp
[cn3:21242] [[15051,0],2] posting recv
[cn3:21242] [[15051,0],2] posting persistent recv on tag 3 for peer [[WILDCARD],WILDCARD]
[cn3:21242] [[15051,0],2] posting recv
[cn3:21242] [[15051,0],2] posting persistent recv on tag 21 for peer [[WILDCARD],WILDCARD]
[cn3:21242] [[15051,0],2] posting recv
[cn3:21242] [[15051,0],2] posting persistent recv on tag 45 for peer [[WILDCARD],WILDCARD]
[cn3:21242] [[15051,0],2] posting recv
[cn3:21242] [[15051,0],2] posting persistent recv on tag 46 for peer [[WILDCARD],WILDCARD]
[cn3:21242] [[15051,0],2] posting recv
[cn3:21242] [[15051,0],2] posting persistent recv on tag 1 for peer [[WILDCARD],WILDCARD]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 10
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2]:tcp:processing set_peer cmd for interface eth0
[cn3:21242] [[15051,0],2] SET_PEER ADDING PEER [[15051,0],0]
[cn3:21242] [[15051,0],2] set_peer: peer [[15051,0],0] is listening on interface eth0 net 192.168.1.1 port 37478
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:10
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:10
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: initiating connection to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],0]
[cn3:21242] [[15051,0],2] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] via interface eth0
[cn3:21242] [[15051,0],2] oob:tcp:peer creating socket to [[15051,0],0]
[cn3:21242] [[15051,0],2] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] via interface eth0 on socket 9
[cn3:21242] [[15051,0],2] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] on 192.168.1.1:37478 - 0 retries
[cn1:13986] [[15051,0],0] mca_oob_tcp_listen_thread: new connection: (17, 11) 192.168.1.3:53152
[cn1:13986] [[15051,0],0] connection_handler: working connection (17, 11) 192.168.1.3:53152
[cn1:13986] [[15051,0],0] accept_connection: 192.168.1.3:53152
[cn1:13986] [[15051,0],0]:tcp:recv:handler called
[cn1:13986] [[15051,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 17
[cn1:13986] [[15051,0],0] waiting for connect ack from UNKNOWN
[cn1:13986] [[15051,0],0] connect ack received from UNKNOWN
[cn1:13986] [[15051,0],0] connect-ack recvd from UNKNOWN
[cn1:13986] [[15051,0],0] mca_oob_tcp_recv_connect: connection from new peer
[cn1:13986] [[15051,0],0] connect-ack header from [[15051,0],2] is okay
[cn1:13986] [[15051,0],0] waiting for connect ack from [[15051,0],2]
[cn1:13986] [[15051,0],0] connect ack received from [[15051,0],2]
[cn1:13986] [[15051,0],0] connect-ack version from [[15051,0],2] matches ours
[cn1:13986] [[15051,0],0] connect-ack [[15051,0],2] authenticated
[cn1:13986] [[15051,0],0] tcp:peer_accept called for peer [[15051,0],2] in state UNKNOWN on socket 17
[cn1:13986] [[15051,0],0] SEND CONNECT ACK
[cn1:13986] [[15051,0],0] send blocking of 47 bytes to socket 17
[cn1:13986] [[15051,0],0] blocking send complete to socket 17
[cn1:13986] [[15051,0],0]-[[15051,0],2] tcp_peer_connected on socket 17
[cn1:13986] [[15051,0],0] routed_binomial_update: [[15051,0],2] --> [[15051,0],2]
[cn1:13986] [[15051,0],0]-[[15051,0],2] accepted: 192.168.1.1 - 192.168.1.3 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:13986] [[15051,0],0] tcp:set_module called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] waiting for connect completion to [[15051,0],0] - activating send event
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_handler CONNECTING
[cn3:21242] [[15051,0],2]:tcp:complete_connect called for peer [[15051,0],0] on socket 9
[cn3:21242] [[15051,0],2] tcp_peer_complete_connect: sending ack to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] SEND CONNECT ACK
[cn3:21242] fetching data from INTERNAL dstore
[cn3:21242] dstore:hash:fetch: searching for key opal.cred on proc 9576316928
[cn3:21242] dstore_hash:fetch key opal.cred for proc 9576316928 not found
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] send blocking of 47 bytes to socket 9
[cn3:21242] [[15051,0],2] blocking send complete to socket 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 96
[cn3:21242] [[15051,0],2] tcp_peer_complete_connect: setting read event on connection to [[15051,0],0]
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] RECV CONNECT ACK FROM [[15051,0],0] ON SOCKET 9
[cn3:21242] [[15051,0],2] waiting for connect ack from [[15051,0],0]
[cn3:21242] [[15051,0],2] connect ack received from [[15051,0],0]
[cn3:21242] [[15051,0],2] connect-ack recvd from [[15051,0],0]
[cn3:21242] [[15051,0],2] connect-ack header from [[15051,0],0] is okay
[cn3:21242] [[15051,0],2] waiting for connect ack from [[15051,0],0]
[cn3:21242] [[15051,0],2] connect ack received from [[15051,0],0]
[cn3:21242] [[15051,0],2] connect-ack version from [[15051,0],0] matches ours
[cn3:21242] [[15051,0],2] connect-ack [[15051,0],0] authenticated
[cn3:21242] [[15051,0],2]-[[15051,0],0] tcp_peer_connected on socket 9
[cn3:21242] [[15051,0],2] routed_binomial_update: [[15051,0],0] --> [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] connected: 192.168.1.3 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn3:21242] [[15051,0],2]:tcp:recv:handler starting send/recv events
[cn3:21242] [[15051,0],2] tcp:set_module called for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 96 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 96 BYTES FOR DEST [[15051,0],0] TAG 10
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 96 bytes from [[15051,0],4] for tag 10
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch from daemon [[15051,0],4]
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch from daemon [[15051,0],4] on node cn5
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch attempting to assign daemon [[15051,0],4] to node cn5
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch completed for daemon [[15051,0],4] at contact 986382336.4;tcp://192.168.1.5:46667
[cn1:13986] [[15051,0],0]: set_addr to uri 986382336.4;tcp://192.168.1.5:46667
[cn1:13986] [[15051,0],0]:set_addr checking if peer [[15051,0],4] is reachable via component tcp
[cn1:13986] [[15051,0],0] oob:tcp: working peer [[15051,0],4] address tcp://192.168.1.5:46667
[cn1:13986] [[15051,0],0] PEER [[15051,0],4] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:13986] [[15051,0],0] PASSING ADDR 192.168.1.5 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:13986] [[15051,0],0]:tcp set addr for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]: peer [[15051,0],4] is reachable via component tcp
[cn1:13986] [[15051,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:13986] [[15051,0],0] set_peer: peer [[15051,0],4] is listening on interface eth0 net 192.168.1.5 port 46667
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 96 BYTES FOR DEST [[15051,0],0] TAG 10
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 96 bytes from [[15051,0],3] for tag 10
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch from daemon [[15051,0],3]
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch from daemon [[15051,0],3] on node cn4
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch attempting to assign daemon [[15051,0],3] to node cn4
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch completed for daemon [[15051,0],3] at contact 986382336.3;tcp://192.168.1.4:54546
[cn1:13986] [[15051,0],0]: set_addr to uri 986382336.3;tcp://192.168.1.4:54546
[cn1:13986] [[15051,0],0]:set_addr checking if peer [[15051,0],3] is reachable via component tcp
[cn1:13986] [[15051,0],0] oob:tcp: working peer [[15051,0],3] address tcp://192.168.1.4:54546
[cn1:13986] [[15051,0],0] PEER [[15051,0],3] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:13986] [[15051,0],0] PASSING ADDR 192.168.1.4 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:13986] [[15051,0],0]:tcp set addr for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]: peer [[15051,0],3] is reachable via component tcp
[cn1:13986] [[15051,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:13986] [[15051,0],0] set_peer: peer [[15051,0],3] is listening on interface eth0 net 192.168.1.4 port 54546
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 96 BYTES FOR DEST [[15051,0],0] TAG 10
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 96 bytes from [[15051,0],2] for tag 10
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch from daemon [[15051,0],2]
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch from daemon [[15051,0],2] on node cn3
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch attempting to assign daemon [[15051,0],2] to node cn3
[cn1:13986] [[15051,0],0] plm:base:orted_report_launch completed for daemon [[15051,0],2] at contact 986382336.2;tcp://192.168.1.3:49500
[cn1:13986] [[15051,0],0]: set_addr to uri 986382336.2;tcp://192.168.1.3:49500
[cn1:13986] [[15051,0],0]:set_addr checking if peer [[15051,0],2] is reachable via component tcp
[cn1:13986] [[15051,0],0] oob:tcp: working peer [[15051,0],2] address tcp://192.168.1.3:49500
[cn1:13986] [[15051,0],0] PEER [[15051,0],2] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:13986] [[15051,0],0] PASSING ADDR 192.168.1.3 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:13986] [[15051,0],0]:tcp set addr for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]: peer [[15051,0],2] is reachable via component tcp
[cn1:13986] [[15051,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:13986] [[15051,0],0] set_peer: peer [[15051,0],2] is listening on interface eth0 net 192.168.1.3 port 49500
[cn1:13986] mca:rmaps: mapping job [15051,1]
[cn1:13986] mca:rmaps: creating new map for job [15051,1]
[cn1:13986] mca:rmaps: nprocs 0
[cn1:13986] mca:rmaps mapping not given - using bycore
[cn1:13986] mca:rmaps:ppr: job [15051,1] not using ppr mapper
[cn1:13986] [[15051,0],0] rmaps:seq mapping job [15051,1]
[cn1:13986] mca:rmaps:seq: job [15051,1] not using seq mapper
[cn1:13986] mca:rmaps:resilient: cannot perform initial map of job [15051,1] - no fault groups
[cn1:13986] mca:rmaps:mindist: job [15051,1] not using mindist mapper
[cn1:13986] mca:rmaps:rr: mapping job [15051,1]
[cn1:13986] [[15051,0],0] Starting with 5 nodes in list
[cn1:13986] [[15051,0],0] Filtering thru apps
[cn1:13986] [[15051,0],0] Retained 5 nodes in list
[cn1:13986] [[15051,0],0] node cn1 has 2 slots available
[cn1:13986] [[15051,0],0] node cn2 has 2 slots available
[cn1:13986] [[15051,0],0] node cn3 has 2 slots available
[cn1:13986] [[15051,0],0] node cn4 has 2 slots available
[cn1:13986] [[15051,0],0] node cn5 has 2 slots available
[cn1:13986] AVAILABLE NODES FOR MAPPING:
[cn1:13986]     node: cn1 daemon: 0
[cn1:13986]     node: cn2 daemon: 1
[cn1:13986]     node: cn3 daemon: 2
[cn1:13986]     node: cn4 daemon: 3
[cn1:13986]     node: cn5 daemon: 4
[cn1:13986] [[15051,0],0] Starting bookmark at node cn1
[cn1:13986] [[15051,0],0] Starting at node cn1
[cn1:13986] mca:rmaps:rr: mapping no-span by Core for job [15051,1] slots 10 num_procs 10
[cn1:13986] mca:rmaps:rr: found 8 Core objects on node cn1
[cn1:13986] mca:rmaps:rr: calculated nprocs 2
[cn1:13986] mca:rmaps:rr: assigning nprocs 2
[cn1:13986] mca:rmaps:rr: assigning proc to object 0
[cn1:13986] mca:rmaps:rr: assigning proc to object 1
[cn1:13986] mca:rmaps:rr: found 8 Core objects on node cn2
[cn1:13986] mca:rmaps:rr: calculated nprocs 2
[cn1:13986] mca:rmaps:rr: assigning nprocs 2
[cn1:13986] mca:rmaps:rr: assigning proc to object 0
[cn1:13986] mca:rmaps:rr: assigning proc to object 1
[cn1:13986] mca:rmaps:rr: found 8 Core objects on node cn3
[cn1:13986] mca:rmaps:rr: calculated nprocs 2
[cn1:13986] mca:rmaps:rr: assigning nprocs 2
[cn1:13986] mca:rmaps:rr: assigning proc to object 0
[cn1:13986] mca:rmaps:rr: assigning proc to object 1
[cn1:13986] mca:rmaps:rr: found 8 Core objects on node cn4
[cn1:13986] mca:rmaps:rr: calculated nprocs 2
[cn1:13986] mca:rmaps:rr: assigning nprocs 2
[cn1:13986] mca:rmaps:rr: assigning proc to object 0
[cn1:13986] mca:rmaps:rr: assigning proc to object 1
[cn1:13986] mca:rmaps:rr: found 8 Core objects on node cn5
[cn1:13986] mca:rmaps:rr: calculated nprocs 2
[cn1:13986] mca:rmaps:rr: assigning nprocs 2
[cn1:13986] mca:rmaps:rr: assigning proc to object 0
[cn1:13986] mca:rmaps:rr: assigning proc to object 1
[cn1:13986] mca:rmaps:base: computing vpids by slot for job [15051,1]
[cn1:13986] mca:rmaps:base: assigning rank 0 to node cn1
[cn1:13986] mca:rmaps:base: assigning rank 1 to node cn1
[cn1:13986] mca:rmaps:base: assigning rank 2 to node cn2
[cn1:13986] mca:rmaps:base: assigning rank 3 to node cn2
[cn1:13986] mca:rmaps:base: assigning rank 4 to node cn3
[cn1:13986] mca:rmaps:base: assigning rank 5 to node cn3
[cn1:13986] mca:rmaps:base: assigning rank 6 to node cn4
[cn1:13986] mca:rmaps:base: assigning rank 7 to node cn4
[cn1:13986] mca:rmaps:base: assigning rank 8 to node cn5
[cn1:13986] mca:rmaps:base: assigning rank 9 to node cn5
[cn1:13986] [[15051,0],0] rmaps:base:compute_usage
[cn1:13986] mca:rmaps: compute bindings for job [15051,1] with policy CORE
[cn1:13986] mca:rmaps: bindings for job [15051,1] - bind in place
[cn1:13986] mca:rmaps: bind in place for job [15051,1] with bindings CORE
[cn1:13986] [[15051,0],0] reset_usage: node cn1 has 2 procs on it
[cn1:13986] [[15051,0],0] reset_usage: ignoring proc [[15051,1],0]
[cn1:13986] [[15051,0],0] reset_usage: ignoring proc [[15051,1],1]
[cn1:13986] BINDING PROC [[15051,1],0] TO Core NUMBER 0
[cn1:13986] [[15051,0],0] BOUND PROC [[15051,1],0] TO 0[Core:0] on node cn1
[cn1:13986] BINDING PROC [[15051,1],1] TO Core NUMBER 1
[cn1:13986] [[15051,0],0] BOUND PROC [[15051,1],1] TO 2[Core:1] on node cn1
[cn1:13986] [[15051,0],0] reset_usage: node cn2 has 2 procs on it
[cn1:13986] [[15051,0],0] reset_usage: ignoring proc [[15051,1],2]
[cn1:13986] [[15051,0],0] reset_usage: ignoring proc [[15051,1],3]
[cn1:13986] BINDING PROC [[15051,1],2] TO Core NUMBER 0
[cn1:13986] [[15051,0],0] BOUND PROC [[15051,1],2] TO 0[Core:0] on node cn2
[cn1:13986] BINDING PROC [[15051,1],3] TO Core NUMBER 1
[cn1:13986] [[15051,0],0] BOUND PROC [[15051,1],3] TO 2[Core:1] on node cn2
[cn1:13986] [[15051,0],0] reset_usage: node cn3 has 2 procs on it
[cn1:13986] [[15051,0],0] reset_usage: ignoring proc [[15051,1],4]
[cn1:13986] [[15051,0],0] reset_usage: ignoring proc [[15051,1],5]
[cn1:13986] BINDING PROC [[15051,1],4] TO Core NUMBER 0
[cn1:13986] [[15051,0],0] BOUND PROC [[15051,1],4] TO 0[Core:0] on node cn3
[cn1:13986] BINDING PROC [[15051,1],5] TO Core NUMBER 1
[cn1:13986] [[15051,0],0] BOUND PROC [[15051,1],5] TO 2[Core:1] on node cn3
[cn1:13986] [[15051,0],0] reset_usage: node cn4 has 2 procs on it
[cn1:13986] [[15051,0],0] reset_usage: ignoring proc [[15051,1],6]
[cn1:13986] [[15051,0],0] reset_usage: ignoring proc [[15051,1],7]
[cn1:13986] BINDING PROC [[15051,1],6] TO Core NUMBER 0
[cn1:13986] [[15051,0],0] BOUND PROC [[15051,1],6] TO 0[Core:0] on node cn4
[cn1:13986] BINDING PROC [[15051,1],7] TO Core NUMBER 1
[cn1:13986] [[15051,0],0] BOUND PROC [[15051,1],7] TO 2[Core:1] on node cn4
[cn1:13986] [[15051,0],0] reset_usage: node cn5 has 2 procs on it
[cn1:13986] [[15051,0],0] reset_usage: ignoring proc [[15051,1],8]
[cn1:13986] [[15051,0],0] reset_usage: ignoring proc [[15051,1],9]
[cn1:13986] BINDING PROC [[15051,1],8] TO Core NUMBER 0
[cn1:13986] [[15051,0],0] BOUND PROC [[15051,1],8] TO 0[Core:0] on node cn5
[cn1:13986] BINDING PROC [[15051,1],9] TO Core NUMBER 1
[cn1:13986] [[15051,0],0] BOUND PROC [[15051,1],9] TO 2[Core:1] on node cn5
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 0 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial 0 found child 1
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 1 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 0
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 1
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 1 parent 0 me 1 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial 1 found child 3
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 3 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 0
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 1
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 1 parent 0 me 3 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 1
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 3
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 3 parent 1 me 3 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 1
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 1
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 0
[cn1:13986] [[15051,0],0] routed:binomial 0 found child 2
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 2 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 0
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 1
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 1 parent 0 me 2 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 1
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 3
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 3 parent 1 me 2 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 3
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 7
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 5
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 2
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 2 parent 0 me 2 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 0
[cn1:13986] [[15051,0],0] routed:binomial 0 found child 4
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 0
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 1
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 1 parent 0 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 1
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 3
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 3 parent 1 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 3
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 7
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 5
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 2
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 2 parent 0 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 2
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 6
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 4
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 4 parent 0 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 0
[cn1:13986] [[15051,0],0]: parent 0 num_children 3
[cn1:13986] [[15051,0],0]: 	child 1
[cn1:13986] [[15051,0],0]: 		relation 3
[cn1:13986] [[15051,0],0]: 	child 2
[cn1:13986] [[15051,0],0]: 	child 4
[cn1:13986] [[15051,0],0] plm:base:launch_apps for job [15051,1]
[cn1:13986] [[15051,0],0] grpcomm:bad:xcast sent to job [15051,0] tag 1
[cn1:13986] [[15051,0],0] rml_send_buffer to peer [[15051,0],0] at tag 15
[cn1:13986] [[15051,0],0] rml_send_msg to peer [[15051,0],0] at tag 15
[cn1:13986] [[15051,0],0] rml_send_iovec_to_self at tag 15
[cn1:13986] [[15051,0],0] rml_send_to_self callback executing for tag 15
[cn1:13986] [[15051,0],0] message received 2634 bytes from [[15051,0],0] for tag 15
[cn1:13986] [[15051,0],0] grpcomm:xcast:recv: with 2634 bytes
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 0 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial 0 found child 1
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 1 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 0
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 1
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 1 parent 0 me 1 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial 1 found child 3
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 3 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 0
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 1
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 1 parent 0 me 3 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 1
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 3
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 3 parent 1 me 3 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 1
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 1
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 0
[cn1:13986] [[15051,0],0] routed:binomial 0 found child 2
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 2 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 0
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 1
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 1 parent 0 me 2 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 1
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 3
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 3 parent 1 me 2 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 3
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 7
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 5
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 2
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 2 parent 0 me 2 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 0
[cn1:13986] [[15051,0],0] routed:binomial 0 found child 4
[cn1:13986] [[15051,0],0] routed:binomial rank 0 parent 0 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 0
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 1
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 1 parent 0 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 1
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 3
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 3 parent 1 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 3
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 7
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 5
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 2
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 2 parent 0 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children of rank 2
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 6
[cn1:13986] [[15051,0],0] routed:binomial find children checking peer 4
[cn1:13986] [[15051,0],0] routed:binomial find children computing tree
[cn1:13986] [[15051,0],0] routed:binomial rank 4 parent 0 me 4 num_procs 5
[cn1:13986] [[15051,0],0] routed:binomial find children returning found value 0
[cn1:13986] [[15051,0],0]: parent 0 num_children 3
[cn1:13986] [[15051,0],0]: 	child 1
[cn1:13986] [[15051,0],0]: 		relation 3
[cn1:13986] [[15051,0],0]: 	child 2
[cn1:13986] [[15051,0],0]: 	child 4
[cn1:13986] [[15051,0],0] routed_binomial: init routes for HNP job [15051,0]
[cn1:13986] [[15051,0],0] rml:base:update:contact:info got uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn1:13986] [[15051,0],0] rml:base:update:contact:info got uri 986382336.1;tcp://192.168.1.2:41722
[cn1:13986] [[15051,0],0] rml:base:update:contact:info got uri 986382336.2;tcp://192.168.1.3:49500
[cn1:13986] [[15051,0],0] rml:base:update:contact:info got uri 986382336.3;tcp://192.168.1.4:54546
[cn1:13986] [[15051,0],0] rml:base:update:contact:info got uri 986382336.4;tcp://192.168.1.5:46667
[cn1:13986] [[15051,0],0] orte:daemon:send_relay sending relay msg to [[15051,0],1]
[cn1:13986] [[15051,0],0] rml_send_buffer to peer [[15051,0],1] at tag 15
[cn1:13986] [[15051,0],0] orte:daemon:send_relay sending relay msg to [[15051,0],2]
[cn1:13986] [[15051,0],0] rml_send_buffer to peer [[15051,0],2] at tag 15
[cn1:13986] [[15051,0],0] orte:daemon:send_relay sending relay msg to [[15051,0],4]
[cn1:13986] [[15051,0],0] rml_send_buffer to peer [[15051,0],4] at tag 15
[cn1:13986] [[15051,0],0] rml_send_buffer to peer [[15051,0],0] at tag 1
[cn1:13986] [[15051,0],0]: set_addr to uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn1:13986] [[15051,0],0]:set_addr peer [[15051,0],0] is me
[cn1:13986] [[15051,0],0]: set_addr to uri 986382336.1;tcp://192.168.1.2:41722
[cn1:13986] [[15051,0],0]:set_addr checking if peer [[15051,0],1] is reachable via component tcp
[cn1:13986] [[15051,0],0] oob:tcp: working peer [[15051,0],1] address tcp://192.168.1.2:41722
[cn1:13986] [[15051,0],0] PEER [[15051,0],1] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:13986] [[15051,0],0] PASSING ADDR 192.168.1.2 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:13986] [[15051,0],0]:tcp set addr for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]: peer [[15051,0],1] is reachable via component tcp
[cn1:13986] [[15051,0],0]: set_addr to uri 986382336.2;tcp://192.168.1.3:49500
[cn1:13986] [[15051,0],0]:set_addr checking if peer [[15051,0],2] is reachable via component tcp
[cn1:13986] [[15051,0],0] oob:tcp: working peer [[15051,0],2] address tcp://192.168.1.3:49500
[cn1:13986] [[15051,0],0] PEER [[15051,0],2] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:13986] [[15051,0],0] PASSING ADDR 192.168.1.3 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:13986] [[15051,0],0]:tcp set addr for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]: peer [[15051,0],2] is reachable via component tcp
[cn1:13986] [[15051,0],0]: set_addr to uri 986382336.3;tcp://192.168.1.4:54546
[cn1:13986] [[15051,0],0]:set_addr checking if peer [[15051,0],3] is reachable via component tcp
[cn1:13986] [[15051,0],0] oob:tcp: working peer [[15051,0],3] address tcp://192.168.1.4:54546
[cn1:13986] [[15051,0],0] PEER [[15051,0],3] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:13986] [[15051,0],0] PASSING ADDR 192.168.1.4 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:13986] [[15051,0],0]:tcp set addr for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]: peer [[15051,0],3] is reachable via component tcp
[cn1:13986] [[15051,0],0]: set_addr to uri 986382336.4;tcp://192.168.1.5:46667
[cn1:13986] [[15051,0],0]:set_addr checking if peer [[15051,0],4] is reachable via component tcp
[cn1:13986] [[15051,0],0] oob:tcp: working peer [[15051,0],4] address tcp://192.168.1.5:46667
[cn1:13986] [[15051,0],0] PEER [[15051,0],4] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:13986] [[15051,0],0] PASSING ADDR 192.168.1.5 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:13986] [[15051,0],0]:tcp set addr for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]: peer [[15051,0],4] is reachable via component tcp
[cn1:13986] [[15051,0],0] rml_send_msg to peer [[15051,0],1] at tag 15
[cn1:13986] [[15051,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] rml_send_msg to peer [[15051,0],2] at tag 15
[cn1:13986] [[15051,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] rml_send_msg to peer [[15051,0],4] at tag 15
[cn1:13986] [[15051,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] rml_send_msg to peer [[15051,0],0] at tag 1
[cn1:13986] [[15051,0],0] rml_send_iovec_to_self at tag 1
[cn1:13986] [[15051,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:13986] [[15051,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:13986] [[15051,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:13986] [[15051,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:13986] [[15051,0],0] oob:base:send to target [[15051,0],1]
[cn1:13986] [[15051,0],0] oob:base:send known transport for peer [[15051,0],1]
[cn1:13986] [[15051,0],0] oob:tcp:send_nb to peer [[15051,0],1]:15
[cn1:13986] [[15051,0],0] tcp:send_nb to peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],1]
[cn1:13986] [[15051,0],0] oob:base:send to target [[15051,0],2]
[cn1:13986] [[15051,0],0] oob:base:send known transport for peer [[15051,0],2]
[cn1:13986] [[15051,0],0] oob:tcp:send_nb to peer [[15051,0],2]:15
[cn1:13986] [[15051,0],0] tcp:send_nb to peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],2]
[cn1:13986] [[15051,0],0] oob:base:send to target [[15051,0],4]
[cn1:13986] [[15051,0],0] oob:base:send known transport for peer [[15051,0],4]
[cn1:13986] [[15051,0],0] oob:tcp:send_nb to peer [[15051,0],4]:15
[cn1:13986] [[15051,0],0] tcp:send_nb to peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],4]
[cn1:13986] [[15051,0],0] rml_send_to_self callback executing for tag 1
[cn1:13986] [[15051,0],0] message received 2634 bytes from [[15051,0],0] for tag 1
[cn1:13986] [[15051,0],0] orted:comm:process_commands() Processing Command: ORTE_DAEMON_ADD_LOCAL_PROCS
[cn1:13986] [[15051,0],0] orted_cmd: received add_local_procs
[cn1:13986] [[15051,0],0] PROGRESSING COLL id 0
[cn1:13986] [[15051,0],0] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn1:13986] [[15051,0],0] PROGRESSING COLL id 1
[cn1:13986] [[15051,0],0] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn1:13986] [[15051,0],0] PROGRESSING COLL id 2
[cn1:13986] [[15051,0],0] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],1]:15
[cn1:13986] [[15051,0],0] routed_binomial_get([[15051,0],1]) --> [[15051,0],1]
[cn1:13986] [[15051,0],0] tcp:send_nb: already connected to [[15051,0],1] - queueing for send
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],1]
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],2]:15
[cn1:13986] [[15051,0],0] routed_binomial_get([[15051,0],2]) --> [[15051,0],2]
[cn1:13986] [[15051,0],0] tcp:send_nb: already connected to [[15051,0],2] - queueing for send
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],2]
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],4]:15
[cn1:13986] [[15051,0],0] routed_binomial_get([[15051,0],4]) --> [[15051,0],4]
[cn1:13986] [[15051,0],0] tcp:send_nb: already connected to [[15051,0],4] - queueing for send
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],4]
[cn1:13986] [[15051,0],0] tcp:send_handler called to send to peer [[15051,0],1]
[cn1:13986] [[15051,0],0] tcp:send_handler SENDING TO [[15051,0],1]
[cn1:13986] [[15051,0],0] MESSAGE SEND COMPLETE TO [[15051,0],1] OF 2634 BYTES ON SOCKET 15
[cn1:13986] [[15051,0],0]-[[15051,0],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] tcp:send_handler called to send to peer [[15051,0],2]
[cn1:13986] [[15051,0],0] tcp:send_handler SENDING TO [[15051,0],2]
[cn1:13986] [[15051,0],0] MESSAGE SEND COMPLETE TO [[15051,0],2] OF 2634 BYTES ON SOCKET 17
[cn1:13986] [[15051,0],0]-[[15051,0],2] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] tcp:send_handler called to send to peer [[15051,0],4]
[cn1:13986] [[15051,0],0] tcp:send_handler SENDING TO [[15051,0],4]
[cn1:13986] [[15051,0],0] MESSAGE SEND COMPLETE TO [[15051,0],4] OF 2634 BYTES ON SOCKET 14
[cn1:13986] [[15051,0],0]-[[15051,0],4] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate data region of size 2634
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:tcp:recv:handler allocate data region of size 2634
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]:tcp:recv:handler allocate data region of size 2634
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] RECVD COMPLETE MESSAGE FROM [[15051,0],0] (ORIGIN [[15051,0],0]) OF 2634 BYTES FOR DEST [[15051,0],1] TAG 15
[cn2:00785] [[15051,0],1] DELIVERING TO RML
[cn2:00785] [[15051,0],1] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] message received 2634 bytes from [[15051,0],0] for tag 15
[cn2:00785] [[15051,0],1] grpcomm:xcast:recv: with 2634 bytes
[cn2:00785] [[15051,0],1] grpcomm:base:xcast updating daemon nidmap
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] RECVD COMPLETE MESSAGE FROM [[15051,0],0] (ORIGIN [[15051,0],0]) OF 2634 BYTES FOR DEST [[15051,0],2] TAG 15
[cn3:21242] [[15051,0],2] DELIVERING TO RML
[cn3:21242] [[15051,0],2] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] message received 2634 bytes from [[15051,0],0] for tag 15
[cn3:21242] [[15051,0],2] grpcomm:xcast:recv: with 2634 bytes
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] RECVD COMPLETE MESSAGE FROM [[15051,0],0] (ORIGIN [[15051,0],0]) OF 2634 BYTES FOR DEST [[15051,0],4] TAG 15
[cn5:13991] [[15051,0],4] DELIVERING TO RML
[cn5:13991] [[15051,0],4] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] message received 2634 bytes from [[15051,0],0] for tag 15
[cn5:13991] [[15051,0],4] grpcomm:xcast:recv: with 2634 bytes
[cn5:13991] [[15051,0],4] grpcomm:base:xcast updating daemon nidmap
[cn2:00785] [[15051,0],1] routed:binomial rank 0 parent 0 me 1 num_procs 5
[cn2:00785] [[15051,0],1] routed:binomial find children of rank 0
[cn2:00785] [[15051,0],1] routed:binomial find children checking peer 1
[cn2:00785] [[15051,0],1] routed:binomial find children computing tree
[cn2:00785] [[15051,0],1] routed:binomial rank 1 parent 0 me 1 num_procs 5
[cn2:00785] [[15051,0],1] routed:binomial 1 found child 3
[cn2:00785] [[15051,0],1] routed:binomial rank 0 parent 0 me 3 num_procs 5
[cn2:00785] [[15051,0],1] routed:binomial find children of rank 0
[cn2:00785] [[15051,0],1] routed:binomial find children checking peer 1
[cn2:00785] [[15051,0],1] routed:binomial find children computing tree
[cn2:00785] [[15051,0],1] routed:binomial rank 1 parent 0 me 3 num_procs 5
[cn2:00785] [[15051,0],1] routed:binomial find children of rank 1
[cn2:00785] [[15051,0],1] routed:binomial find children checking peer 3
[cn2:00785] [[15051,0],1] routed:binomial find children computing tree
[cn2:00785] [[15051,0],1] routed:binomial rank 3 parent 1 me 3 num_procs 5
[cn2:00785] [[15051,0],1] routed:binomial find children returning found value 1
[cn2:00785] [[15051,0],1] routed:binomial find children returning found value 1
[cn2:00785] [[15051,0],1] routed:binomial find children returning found value 0
[cn2:00785] [[15051,0],1]: parent 0 num_children 1
[cn2:00785] [[15051,0],1]: 	child 3
[cn2:00785] [[15051,0],1] routed_binomial: init routes for daemon job [15051,0]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn2:00785] [[15051,0],1] rml:base:update:contact:info got uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn2:00785] [[15051,0],1] rml:base:update:contact:info got uri 986382336.1;tcp://192.168.1.2:41722
[cn2:00785] [[15051,0],1] rml:base:update:contact:info got uri 986382336.2;tcp://192.168.1.3:49500
[cn2:00785] [[15051,0],1] rml:base:update:contact:info got uri 986382336.3;tcp://192.168.1.4:54546
[cn2:00785] [[15051,0],1] rml:base:update:contact:info got uri 986382336.4;tcp://192.168.1.5:46667
[cn2:00785] [[15051,0],1] orte:daemon:send_relay sending relay msg to [[15051,0],3]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],3] at tag 15
[cn3:21242] [[15051,0],2] grpcomm:base:xcast updating daemon nidmap
[cn3:21242] [[15051,0],2] routed:binomial rank 0 parent 0 me 2 num_procs 5
[cn3:21242] [[15051,0],2] routed:binomial find children of rank 0
[cn3:21242] [[15051,0],2] routed:binomial find children checking peer 1
[cn3:21242] [[15051,0],2] routed:binomial find children computing tree
[cn3:21242] [[15051,0],2] routed:binomial rank 1 parent 0 me 2 num_procs 5
[cn3:21242] [[15051,0],2] routed:binomial find children of rank 1
[cn3:21242] [[15051,0],2] routed:binomial find children checking peer 3
[cn3:21242] [[15051,0],2] routed:binomial find children computing tree
[cn3:21242] [[15051,0],2] routed:binomial rank 3 parent 1 me 2 num_procs 5
[cn3:21242] [[15051,0],2] routed:binomial find children of rank 3
[cn3:21242] [[15051,0],2] routed:binomial find children checking peer 7
[cn3:21242] [[15051,0],2] routed:binomial find children checking peer 5
[cn3:21242] [[15051,0],2] routed:binomial find children checking peer 2
[cn3:21242] [[15051,0],2] routed:binomial find children computing tree
[cn3:21242] [[15051,0],2] routed:binomial rank 2 parent 0 me 2 num_procs 5
[cn3:21242] [[15051,0],2] routed:binomial find children returning found value 0
[cn3:21242] [[15051,0],2]: parent 0 num_children 0
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],1] at tag 1
[cn2:00785] [[15051,0],1]: set_addr to uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn2:00785] [[15051,0],1]:set_addr checking if peer [[15051,0],0] is reachable via component tcp
[cn2:00785] [[15051,0],1] oob:tcp: working peer [[15051,0],0] address tcp://192.168.1.1,192.168.10.2:37478
[cn2:00785] [[15051,0],1] PEER [[15051,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn2:00785] [[15051,0],1] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00785] [[15051,0],1]:tcp set addr for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn2:00785] [[15051,0],1]: peer [[15051,0],0] is reachable via component tcp
[cn2:00785] [[15051,0],1]: set_addr to uri 986382336.1;tcp://192.168.1.2:41722
[cn5:13991] [[15051,0],4] routed:binomial rank 0 parent 0 me 4 num_procs 5
[cn5:13991] [[15051,0],4] routed:binomial find children of rank 0
[cn5:13991] [[15051,0],4] routed:binomial find children checking peer 1
[cn5:13991] [[15051,0],4] routed:binomial find children computing tree
[cn5:13991] [[15051,0],4] routed:binomial rank 1 parent 0 me 4 num_procs 5
[cn5:13991] [[15051,0],4] routed:binomial find children of rank 1
[cn5:13991] [[15051,0],4] routed:binomial find children checking peer 3
[cn5:13991] [[15051,0],4] routed:binomial find children computing tree
[cn5:13991] [[15051,0],4] routed:binomial rank 3 parent 1 me 4 num_procs 5
[cn5:13991] [[15051,0],4] routed:binomial find children of rank 3
[cn5:13991] [[15051,0],4] routed:binomial find children checking peer 7
[cn5:13991] [[15051,0],4] routed:binomial find children checking peer 5
[cn5:13991] [[15051,0],4] routed:binomial find children checking peer 2
[cn5:13991] [[15051,0],4] routed:binomial find children computing tree
[cn2:00785] [[15051,0],1]:set_addr peer [[15051,0],1] is me
[cn2:00785] [[15051,0],1]: set_addr to uri 986382336.2;tcp://192.168.1.3:49500
[cn5:13991] [[15051,0],4] routed:binomial rank 2 parent 0 me 4 num_procs 5
[cn5:13991] [[15051,0],4] routed:binomial find children of rank 2
[cn5:13991] [[15051,0],4] routed:binomial find children checking peer 6
[cn5:13991] [[15051,0],4] routed:binomial find children checking peer 4
[cn5:13991] [[15051,0],4] routed:binomial find children computing tree
[cn5:13991] [[15051,0],4] routed:binomial rank 4 parent 0 me 4 num_procs 5
[cn5:13991] [[15051,0],4] routed:binomial find children returning found value 0
[cn5:13991] [[15051,0],4]: parent 0 num_children 0
[cn5:13991] [[15051,0],4] routed_binomial: init routes for daemon job [15051,0]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn2:00785] [[15051,0],1]:set_addr checking if peer [[15051,0],2] is reachable via component tcp
[cn2:00785] [[15051,0],1] oob:tcp: working peer [[15051,0],2] address tcp://192.168.1.3:49500
[cn3:21242] [[15051,0],2] routed_binomial: init routes for daemon job [15051,0]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn3:21242] [[15051,0],2] rml:base:update:contact:info got uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn3:21242] [[15051,0],2] rml:base:update:contact:info got uri 986382336.1;tcp://192.168.1.2:41722
[cn3:21242] [[15051,0],2] rml:base:update:contact:info got uri 986382336.2;tcp://192.168.1.3:49500
[cn3:21242] [[15051,0],2] rml:base:update:contact:info got uri 986382336.3;tcp://192.168.1.4:54546
[cn3:21242] [[15051,0],2] rml:base:update:contact:info got uri 986382336.4;tcp://192.168.1.5:46667
[cn5:13991] [[15051,0],4] rml:base:update:contact:info got uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn5:13991] [[15051,0],4] rml:base:update:contact:info got uri 986382336.1;tcp://192.168.1.2:41722
[cn5:13991] [[15051,0],4] rml:base:update:contact:info got uri 986382336.2;tcp://192.168.1.3:49500
[cn5:13991] [[15051,0],4] rml:base:update:contact:info got uri 986382336.3;tcp://192.168.1.4:54546
[cn5:13991] [[15051,0],4] rml:base:update:contact:info got uri 986382336.4;tcp://192.168.1.5:46667
[cn5:13991] [[15051,0],4] orte:daemon:send_relay - recipient list is empty!
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],4] at tag 1
[cn5:13991] [[15051,0],4]: set_addr to uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn2:00785] [[15051,0],1] PEER [[15051,0],2] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn2:00785] [[15051,0],1] PASSING ADDR 192.168.1.3 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21242] [[15051,0],2] orte:daemon:send_relay - recipient list is empty!
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],2] at tag 1
[cn3:21242] [[15051,0],2]: set_addr to uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn3:21242] [[15051,0],2]:set_addr checking if peer [[15051,0],0] is reachable via component tcp
[cn5:13991] [[15051,0],4]:set_addr checking if peer [[15051,0],0] is reachable via component tcp
[cn5:13991] [[15051,0],4] oob:tcp: working peer [[15051,0],0] address tcp://192.168.1.1,192.168.10.2:37478
[cn2:00785] [[15051,0],1]:tcp set addr for peer [[15051,0],2]
[cn2:00785] [[15051,0],1]: peer [[15051,0],2] is reachable via component tcp
[cn3:21242] [[15051,0],2] oob:tcp: working peer [[15051,0],0] address tcp://192.168.1.1,192.168.10.2:37478
[cn5:13991] [[15051,0],4] PEER [[15051,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn2:00785] [[15051,0],1]: set_addr to uri 986382336.3;tcp://192.168.1.4:54546
[cn2:00785] [[15051,0],1]:set_addr checking if peer [[15051,0],3] is reachable via component tcp
[cn3:21242] [[15051,0],2] PEER [[15051,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:13991] [[15051,0],4] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn5:13991] [[15051,0],4]:tcp set addr for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp: working peer [[15051,0],3] address tcp://192.168.1.4:54546
[cn3:21242] [[15051,0],2] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21242] [[15051,0],2]:tcp set addr for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn2:00785] [[15051,0],1] PEER [[15051,0],3] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn3:21242] [[15051,0],2] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn3:21242] [[15051,0],2]: peer [[15051,0],0] is reachable via component tcp
[cn3:21242] [[15051,0],2]: set_addr to uri 986382336.1;tcp://192.168.1.2:41722
[cn5:13991] [[15051,0],4]: peer [[15051,0],0] is reachable via component tcp
[cn5:13991] [[15051,0],4]: set_addr to uri 986382336.1;tcp://192.168.1.2:41722
[cn2:00785] [[15051,0],1] PASSING ADDR 192.168.1.4 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00785] [[15051,0],1]:tcp set addr for peer [[15051,0],3]
[cn3:21242] [[15051,0],2]:set_addr checking if peer [[15051,0],1] is reachable via component tcp
[cn5:13991] [[15051,0],4]:set_addr checking if peer [[15051,0],1] is reachable via component tcp
[cn5:13991] [[15051,0],4] oob:tcp: working peer [[15051,0],1] address tcp://192.168.1.2:41722
[cn2:00785] [[15051,0],1]: peer [[15051,0],3] is reachable via component tcp
[cn2:00785] [[15051,0],1]: set_addr to uri 986382336.4;tcp://192.168.1.5:46667
[cn2:00785] [[15051,0],1]:set_addr checking if peer [[15051,0],4] is reachable via component tcp
[cn3:21242] [[15051,0],2] oob:tcp: working peer [[15051,0],1] address tcp://192.168.1.2:41722
[cn5:13991] [[15051,0],4] PEER [[15051,0],1] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:13991] [[15051,0],4] PASSING ADDR 192.168.1.2 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00785] [[15051,0],1] oob:tcp: working peer [[15051,0],4] address tcp://192.168.1.5:46667
[cn3:21242] [[15051,0],2] PEER [[15051,0],1] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:13991] [[15051,0],4]:tcp set addr for peer [[15051,0],1]
[cn5:13991] [[15051,0],4]: peer [[15051,0],1] is reachable via component tcp
[cn2:00785] [[15051,0],1] PEER [[15051,0],4] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn3:21242] [[15051,0],2] PASSING ADDR 192.168.1.2 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21242] [[15051,0],2]:tcp set addr for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] PASSING ADDR 192.168.1.5 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00785] [[15051,0],1]:tcp set addr for peer [[15051,0],4]
[cn3:21242] [[15051,0],2]: peer [[15051,0],1] is reachable via component tcp
[cn3:21242] [[15051,0],2]: set_addr to uri 986382336.2;tcp://192.168.1.3:49500
[cn2:00785] [[15051,0],1]: peer [[15051,0],4] is reachable via component tcp
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],3] at tag 15
[cn3:21242] [[15051,0],2]:set_addr peer [[15051,0],2] is me
[cn3:21242] [[15051,0],2]: set_addr to uri 986382336.3;tcp://192.168.1.4:54546
[cn3:21242] [[15051,0],2]:set_addr checking if peer [[15051,0],3] is reachable via component tcp
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],1] at tag 1
[cn2:00785] [[15051,0],1] rml_send_iovec_to_self at tag 1
[cn3:21242] [[15051,0],2] oob:tcp: working peer [[15051,0],3] address tcp://192.168.1.4:54546
[cn2:00785] [[15051,0],1]:tcp:processing set_peer cmd for interface eth0
[cn2:00785] [[15051,0],1]:tcp:processing set_peer cmd for interface eth0
[cn3:21242] [[15051,0],2] PEER [[15051,0],3] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn2:00785] [[15051,0],1] SET_PEER ADDING PEER [[15051,0],2]
[cn2:00785] [[15051,0],1] set_peer: peer [[15051,0],2] is listening on interface eth0 net 192.168.1.3 port 49500
[cn3:21242] [[15051,0],2] PASSING ADDR 192.168.1.4 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21242] [[15051,0],2]:tcp set addr for peer [[15051,0],3]
[cn2:00785] [[15051,0],1]:tcp:processing set_peer cmd for interface eth0
[cn2:00785] [[15051,0],1] SET_PEER ADDING PEER [[15051,0],3]
[cn2:00785] [[15051,0],1] set_peer: peer [[15051,0],3] is listening on interface eth0 net 192.168.1.4 port 54546
[cn3:21242] [[15051,0],2]: peer [[15051,0],3] is reachable via component tcp
[cn3:21242] [[15051,0],2]: set_addr to uri 986382336.4;tcp://192.168.1.5:46667
[cn2:00785] [[15051,0],1]:tcp:processing set_peer cmd for interface eth0
[cn2:00785] [[15051,0],1] SET_PEER ADDING PEER [[15051,0],4]
[cn2:00785] [[15051,0],1] set_peer: peer [[15051,0],4] is listening on interface eth0 net 192.168.1.5 port 46667
[cn3:21242] [[15051,0],2]:set_addr checking if peer [[15051,0],4] is reachable via component tcp
[cn3:21242] [[15051,0],2] oob:tcp: working peer [[15051,0],4] address tcp://192.168.1.5:46667
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],3]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],3]:15
[cn3:21242] [[15051,0],2] PEER [[15051,0],4] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn3:21242] [[15051,0],2] PASSING ADDR 192.168.1.5 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21242] [[15051,0],2]:tcp set addr for peer [[15051,0],4]
[cn3:21242] [[15051,0],2]: peer [[15051,0],4] is reachable via component tcp
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],2] at tag 1
[cn3:21242] [[15051,0],2] rml_send_iovec_to_self at tag 1
[cn5:13991] [[15051,0],4]: set_addr to uri 986382336.2;tcp://192.168.1.3:49500
[cn5:13991] [[15051,0],4]:set_addr checking if peer [[15051,0],2] is reachable via component tcp
[cn5:13991] [[15051,0],4] oob:tcp: working peer [[15051,0],2] address tcp://192.168.1.3:49500
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],3]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],3]
[cn2:00785] [[15051,0],1] rml_send_to_self callback executing for tag 1
[cn5:13991] [[15051,0],4] PEER [[15051,0],2] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn2:00785] [[15051,0],1] message received 2634 bytes from [[15051,0],1] for tag 1
[cn2:00785] [[15051,0],1] orted:comm:process_commands() Processing Command: ORTE_DAEMON_ADD_LOCAL_PROCS
[cn5:13991] [[15051,0],4] PASSING ADDR 192.168.1.3 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn5:13991] [[15051,0],4]:tcp set addr for peer [[15051,0],2]
[cn2:00785] [[15051,0],1] orted_cmd: received add_local_procs
[cn5:13991] [[15051,0],4]: peer [[15051,0],2] is reachable via component tcp
[cn5:13991] [[15051,0],4]: set_addr to uri 986382336.3;tcp://192.168.1.4:54546
[cn2:00785] [[15051,0],1] PROGRESSING COLL id 0
[cn2:00785] [[15051,0],1] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn2:00785] [[15051,0],1] PROGRESSING COLL id 1
[cn2:00785] [[15051,0],1] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn2:00785] [[15051,0],1] PROGRESSING COLL id 2
[cn2:00785] [[15051,0],1] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],3]:15
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],3]) --> [[15051,0],3]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],3]
[cn2:00785] [[15051,0],1] tcp:send_nb: initiating connection to [[15051,0],3]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],3]
[cn5:13991] [[15051,0],4]:set_addr checking if peer [[15051,0],3] is reachable via component tcp
[cn5:13991] [[15051,0],4] oob:tcp: working peer [[15051,0],3] address tcp://192.168.1.4:54546
[cn2:00785] [[15051,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],3] via interface eth0
[cn2:00785] [[15051,0],1] oob:tcp:peer creating socket to [[15051,0],3]
[cn2:00785] [[15051,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],3] via interface eth0 on socket 10
[cn2:00785] [[15051,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],3] on 192.168.1.4:54546 - 0 retries
[cn5:13991] [[15051,0],4] PEER [[15051,0],3] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:13991] [[15051,0],4] PASSING ADDR 192.168.1.4 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn5:13991] [[15051,0],4]:tcp set addr for peer [[15051,0],3]
[cn5:13991] [[15051,0],4]: peer [[15051,0],3] is reachable via component tcp
[cn5:13991] [[15051,0],4]: set_addr to uri 986382336.4;tcp://192.168.1.5:46667
[cn5:13991] [[15051,0],4]:set_addr peer [[15051,0],4] is me
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],4] at tag 1
[cn5:13991] [[15051,0],4] rml_send_iovec_to_self at tag 1
[cn5:13991] [[15051,0],4]:tcp:processing set_peer cmd for interface eth0
[cn5:13991] [[15051,0],4]:tcp:processing set_peer cmd for interface eth0
[cn5:13991] [[15051,0],4] SET_PEER ADDING PEER [[15051,0],1]
[cn5:13991] [[15051,0],4] set_peer: peer [[15051,0],1] is listening on interface eth0 net 192.168.1.2 port 41722
[cn5:13991] [[15051,0],4]:tcp:processing set_peer cmd for interface eth0
[cn5:13991] [[15051,0],4] SET_PEER ADDING PEER [[15051,0],2]
[cn5:13991] [[15051,0],4] set_peer: peer [[15051,0],2] is listening on interface eth0 net 192.168.1.3 port 49500
[cn3:21242] [[15051,0],2]:tcp:processing set_peer cmd for interface eth0
[cn3:21242] [[15051,0],2]:tcp:processing set_peer cmd for interface eth0
[cn3:21242] [[15051,0],2] SET_PEER ADDING PEER [[15051,0],1]
[cn3:21242] [[15051,0],2] set_peer: peer [[15051,0],1] is listening on interface eth0 net 192.168.1.2 port 41722
[cn3:21242] [[15051,0],2]:tcp:processing set_peer cmd for interface eth0
[cn3:21242] [[15051,0],2] SET_PEER ADDING PEER [[15051,0],3]
[cn3:21242] [[15051,0],2] set_peer: peer [[15051,0],3] is listening on interface eth0 net 192.168.1.4 port 54546
[cn3:21242] [[15051,0],2]:tcp:processing set_peer cmd for interface eth0
[cn3:21242] [[15051,0],2] SET_PEER ADDING PEER [[15051,0],4]
[cn3:21242] [[15051,0],2] set_peer: peer [[15051,0],4] is listening on interface eth0 net 192.168.1.5 port 46667
[cn3:21242] [[15051,0],2] rml_send_to_self callback executing for tag 1
[cn3:21242] [[15051,0],2] message received 2634 bytes from [[15051,0],2] for tag 1
[cn3:21242] [[15051,0],2] orted:comm:process_commands() Processing Command: ORTE_DAEMON_ADD_LOCAL_PROCS
[cn3:21242] [[15051,0],2] orted_cmd: received add_local_procs
[cn5:13991] [[15051,0],4]:tcp:processing set_peer cmd for interface eth0
[cn5:13991] [[15051,0],4] SET_PEER ADDING PEER [[15051,0],3]
[cn5:13991] [[15051,0],4] set_peer: peer [[15051,0],3] is listening on interface eth0 net 192.168.1.4 port 54546
[cn5:13991] [[15051,0],4] rml_send_to_self callback executing for tag 1
[cn2:00785] [[15051,0],1] waiting for connect completion to [[15051,0],3] - activating send event
[cn5:13991] [[15051,0],4] message received 2634 bytes from [[15051,0],4] for tag 1
[cn5:13991] [[15051,0],4] orted:comm:process_commands() Processing Command: ORTE_DAEMON_ADD_LOCAL_PROCS
[cn5:13991] [[15051,0],4] orted_cmd: received add_local_procs
[cn5:13991] [[15051,0],4] PROGRESSING COLL id 0
[cn5:13991] [[15051,0],4] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn5:13991] [[15051,0],4] PROGRESSING COLL id 1
[cn5:13991] [[15051,0],4] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn5:13991] [[15051,0],4] PROGRESSING COLL id 2
[cn5:13991] [[15051,0],4] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn4:17654] [[15051,0],3] connection_event_handler: working connection (10, 115) 192.168.1.2:54610
[cn4:17654] [[15051,0],3] accept_connection: 192.168.1.2:54610
[cn3:21242] [[15051,0],2] PROGRESSING COLL id 0
[cn3:21242] [[15051,0],2] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn3:21242] [[15051,0],2] PROGRESSING COLL id 1
[cn3:21242] [[15051,0],2] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn3:21242] [[15051,0],2] PROGRESSING COLL id 2
[cn3:21242] [[15051,0],2] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 5
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 5
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],3]
[cn2:00785] [[15051,0],1] tcp:send_handler CONNECTING
[cn2:00785] [[15051,0],1]:tcp:complete_connect called for peer [[15051,0],3] on socket 10
[cn2:00785] [[15051,0],1] tcp_peer_complete_connect: sending ack to [[15051,0],3]
[cn2:00785] [[15051,0],1] SEND CONNECT ACK
[cn2:00785] [[15051,0],1] send blocking of 47 bytes to socket 10
[cn2:00785] [[15051,0],1] blocking send complete to socket 10
[cn2:00785] [[15051,0],1] tcp_peer_complete_connect: setting read event on connection to [[15051,0],3]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 5
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 5
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:5
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:5
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:tcp:recv:handler called
[cn4:17654] [[15051,0],3] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 10
[cn4:17654] [[15051,0],3] waiting for connect ack from UNKNOWN
[cn4:17654] [[15051,0],3] connect ack received from UNKNOWN
[cn4:17654] [[15051,0],3] connect-ack recvd from UNKNOWN
[cn4:17654] [[15051,0],3] mca_oob_tcp_recv_connect: connection from new peer
[cn4:17654] [[15051,0],3] connect-ack header from [[15051,0],1] is okay
[cn4:17654] [[15051,0],3] waiting for connect ack from [[15051,0],1]
[cn4:17654] [[15051,0],3] connect ack received from [[15051,0],1]
[cn4:17654] [[15051,0],3] connect-ack version from [[15051,0],1] matches ours
[cn4:17654] [[15051,0],3] connect-ack [[15051,0],1] authenticated
[cn4:17654] [[15051,0],3] tcp:peer_accept called for peer [[15051,0],1] in state UNKNOWN on socket 10
[cn4:17654] [[15051,0],3] SEND CONNECT ACK
[cn4:17654] [[15051,0],3] send blocking of 47 bytes to socket 10
[cn4:17654] [[15051,0],3] blocking send complete to socket 10
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:5
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:5
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 5
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 5
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:5
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:5
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3]-[[15051,0],1] tcp_peer_connected on socket 10
[cn4:17654] [[15051,0],3] routed_binomial_update: [[15051,0],1] --> [[15051,0],1]
[cn4:17654] [[15051,0],3]-[[15051,0],1] accepted: 192.168.1.4 - 192.168.1.2 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn4:17654] [[15051,0],3] tcp:set_module called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,0],3]
[cn2:00785] [[15051,0],1] RECV CONNECT ACK FROM [[15051,0],3] ON SOCKET 10
[cn2:00785] [[15051,0],1] waiting for connect ack from [[15051,0],3]
[cn2:00785] [[15051,0],1] connect ack received from [[15051,0],3]
[cn2:00785] [[15051,0],1] connect-ack recvd from [[15051,0],3]
[cn2:00785] [[15051,0],1] connect-ack header from [[15051,0],3] is okay
[cn2:00785] [[15051,0],1] waiting for connect ack from [[15051,0],3]
[cn2:00785] [[15051,0],1] connect ack received from [[15051,0],3]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] connect-ack version from [[15051,0],3] matches ours
[cn2:00785] [[15051,0],1] connect-ack [[15051,0],3] authenticated
[cn2:00785] [[15051,0],1]-[[15051,0],3] tcp_peer_connected on socket 10
[cn2:00785] [[15051,0],1] routed_binomial_update: [[15051,0],3] --> [[15051,0],3]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],3] connected: 192.168.1.2 - 192.168.1.4 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn2:00785] [[15051,0],1]:tcp:recv:handler starting send/recv events
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:set_module called for peer [[15051,0],3]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],3]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],3]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],3] OF 2634 BYTES ON SOCKET 10
[cn2:00785] [[15051,0],1]-[[15051,0],3] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,0],1]
[cn4:17654] [[15051,0],3]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]:tcp:recv:handler allocate data region of size 2634
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,0],1]
[cn4:17654] [[15051,0],3]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 2634 BYTES FOR DEST [[15051,0],3] TAG 15
[cn4:17654] [[15051,0],3] DELIVERING TO RML
[cn4:17654] [[15051,0],3] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] message received 2634 bytes from [[15051,0],1] for tag 15
[cn4:17654] [[15051,0],3] grpcomm:xcast:recv: with 2634 bytes
[cn4:17654] [[15051,0],3] grpcomm:base:xcast updating daemon nidmap
[cn4:17654] [[15051,0],3] routed:binomial rank 0 parent 0 me 3 num_procs 5
[cn4:17654] [[15051,0],3] routed:binomial find children of rank 0
[cn4:17654] [[15051,0],3] routed:binomial find children checking peer 1
[cn4:17654] [[15051,0],3] routed:binomial find children computing tree
[cn4:17654] [[15051,0],3] routed:binomial rank 1 parent 0 me 3 num_procs 5
[cn4:17654] [[15051,0],3] routed:binomial find children of rank 1
[cn4:17654] [[15051,0],3] routed:binomial find children checking peer 3
[cn4:17654] [[15051,0],3] routed:binomial find children computing tree
[cn4:17654] [[15051,0],3] routed:binomial rank 3 parent 1 me 3 num_procs 5
[cn4:17654] [[15051,0],3] routed:binomial find children returning found value 1
[cn4:17654] [[15051,0],3] routed:binomial find children returning found value 1
[cn4:17654] [[15051,0],3]: parent 1 num_children 0
[cn4:17654] [[15051,0],3] routed_binomial: init routes for daemon job [15051,0]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn4:17654] [[15051,0],3] rml:base:update:contact:info got uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn4:17654] [[15051,0],3] rml:base:update:contact:info got uri 986382336.1;tcp://192.168.1.2:41722
[cn4:17654] [[15051,0],3] rml:base:update:contact:info got uri 986382336.2;tcp://192.168.1.3:49500
[cn4:17654] [[15051,0],3] rml:base:update:contact:info got uri 986382336.3;tcp://192.168.1.4:54546
[cn4:17654] [[15051,0],3] rml:base:update:contact:info got uri 986382336.4;tcp://192.168.1.5:46667
[cn4:17654] [[15051,0],3] orte:daemon:send_relay - recipient list is empty!
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],3] at tag 1
[cn4:17654] [[15051,0],3]: set_addr to uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn4:17654] [[15051,0],3]:set_addr checking if peer [[15051,0],0] is reachable via component tcp
[cn4:17654] [[15051,0],3] oob:tcp: working peer [[15051,0],0] address tcp://192.168.1.1,192.168.10.2:37478
[cn4:17654] [[15051,0],3] PEER [[15051,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17654] [[15051,0],3] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn4:17654] [[15051,0],3]:tcp set addr for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn4:17654] [[15051,0],3]: peer [[15051,0],0] is reachable via component tcp
[cn4:17654] [[15051,0],3]: set_addr to uri 986382336.1;tcp://192.168.1.2:41722
[cn4:17654] [[15051,0],3]:set_addr checking if peer [[15051,0],1] is reachable via component tcp
[cn4:17654] [[15051,0],3] oob:tcp: working peer [[15051,0],1] address tcp://192.168.1.2:41722
[cn4:17654] [[15051,0],3] PEER [[15051,0],1] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17654] [[15051,0],3] PASSING ADDR 192.168.1.2 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn4:17654] [[15051,0],3]:tcp set addr for peer [[15051,0],1]
[cn4:17654] [[15051,0],3]: peer [[15051,0],1] is reachable via component tcp
[cn4:17654] [[15051,0],3]: set_addr to uri 986382336.2;tcp://192.168.1.3:49500
[cn4:17654] [[15051,0],3]:set_addr checking if peer [[15051,0],2] is reachable via component tcp
[cn4:17654] [[15051,0],3] oob:tcp: working peer [[15051,0],2] address tcp://192.168.1.3:49500
[cn4:17654] [[15051,0],3] PEER [[15051,0],2] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17654] [[15051,0],3] PASSING ADDR 192.168.1.3 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn4:17654] [[15051,0],3]:tcp set addr for peer [[15051,0],2]
[cn4:17654] [[15051,0],3]: peer [[15051,0],2] is reachable via component tcp
[cn4:17654] [[15051,0],3]: set_addr to uri 986382336.3;tcp://192.168.1.4:54546
[cn4:17654] [[15051,0],3]:set_addr peer [[15051,0],3] is me
[cn4:17654] [[15051,0],3]: set_addr to uri 986382336.4;tcp://192.168.1.5:46667
[cn4:17654] [[15051,0],3]:set_addr checking if peer [[15051,0],4] is reachable via component tcp
[cn4:17654] [[15051,0],3] oob:tcp: working peer [[15051,0],4] address tcp://192.168.1.5:46667
[cn4:17654] [[15051,0],3] PEER [[15051,0],4] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17654] [[15051,0],3] PASSING ADDR 192.168.1.5 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn4:17654] [[15051,0],3]:tcp set addr for peer [[15051,0],4]
[cn4:17654] [[15051,0],3]: peer [[15051,0],4] is reachable via component tcp
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],3] at tag 1
[cn4:17654] [[15051,0],3] rml_send_iovec_to_self at tag 1
[cn4:17654] [[15051,0],3]:tcp:processing set_peer cmd for interface eth0
[cn4:17654] [[15051,0],3]:tcp:processing set_peer cmd for interface eth0
[cn4:17654] [[15051,0],3] set_peer: peer [[15051,0],1] is listening on interface eth0 net 192.168.1.2 port 41722
[cn4:17654] [[15051,0],3]:tcp:processing set_peer cmd for interface eth0
[cn4:17654] [[15051,0],3] SET_PEER ADDING PEER [[15051,0],2]
[cn4:17654] [[15051,0],3] set_peer: peer [[15051,0],2] is listening on interface eth0 net 192.168.1.3 port 49500
[cn4:17654] [[15051,0],3]:tcp:processing set_peer cmd for interface eth0
[cn4:17654] [[15051,0],3] SET_PEER ADDING PEER [[15051,0],4]
[cn4:17654] [[15051,0],3] set_peer: peer [[15051,0],4] is listening on interface eth0 net 192.168.1.5 port 46667
[cn4:17654] [[15051,0],3] rml_send_to_self callback executing for tag 1
[cn4:17654] [[15051,0],3] message received 2634 bytes from [[15051,0],3] for tag 1
[cn4:17654] [[15051,0],3] orted:comm:process_commands() Processing Command: ORTE_DAEMON_ADD_LOCAL_PROCS
[cn4:17654] [[15051,0],3] orted_cmd: received add_local_procs
[cn4:17654] [[15051,0],3] PROGRESSING COLL id 0
[cn4:17654] [[15051,0],3] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn4:17654] [[15051,0],3] PROGRESSING COLL id 1
[cn4:17654] [[15051,0],3] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn4:17654] [[15051,0],3] PROGRESSING COLL id 2
[cn4:17654] [[15051,0],3] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 311 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 81 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 81 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 90 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 260 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 80 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 311 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 87 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 93 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 311 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 5
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 5
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:5
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:5
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 81 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 260 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 121 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 93 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 97 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 109 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 291 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 76 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 263 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 121 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 500 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 546 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 76 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 121 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 592 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 93 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 217 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 437 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 76 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 192 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 546 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 311 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 311 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 247 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 271 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 159 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 482 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 136 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 401 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 88 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 247 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 263 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 500 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 442 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 76 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 263 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 172 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 304 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 263 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 304 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 192 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 401 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 88 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 172 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 546 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 353 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 88 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 78 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 187 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 500 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 247 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 373 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 172 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 401 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 401 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 88 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 537 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 88 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 263 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 276 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 304 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 247 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 245 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 370 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 756 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 156 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 442 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 794 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 247 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 276 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 263 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 443 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1313 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 164 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 304 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 104 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 192 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 102 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 304 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 264 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 286 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 118 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 96 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 220 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 105 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 192 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 104 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 738 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 104 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 190 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 202 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 202 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 443 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1256 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 443 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 69 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 221 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 543 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1256 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 622 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 221 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 176 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 104 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1251 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 264 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 286 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 374 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 419 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 106 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 121 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 206 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 104 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 126 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 711 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 443 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 106 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1256 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 766 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 267 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1019 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 221 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 738 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] connection_event_handler: working connection (12, 0) 192.168.1.5:56530
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] accept_connection: 192.168.1.5:56530
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 190 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1291 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 126 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]:tcp:recv:handler called
[cn5:13991] [[15051,0],4] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 12
[cn5:13991] [[15051,0],4] waiting for connect ack from UNKNOWN
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] connect ack received from UNKNOWN
[cn5:13991] [[15051,0],4] connect-ack recvd from UNKNOWN
[cn5:13991] [[15051,0],4] mca_oob_tcp_recv_connect: connection from new peer
[cn5:13991] [[15051,0],4] connect-ack header from [[15051,1],8] is okay
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4] waiting for connect ack from [[15051,1],8]
[cn5:13991] [[15051,0],4] connect ack received from [[15051,1],8]
[cn5:13991] [[15051,0],4] connect-ack version from [[15051,1],8] matches ours
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 106 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] connect-ack [[15051,1],8] authenticated
[cn5:13991] [[15051,0],4] tcp:peer_accept called for peer [[15051,1],8] in state UNKNOWN on socket 12
[cn5:13991] [[15051,0],4] SEND CONNECT ACK
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] send blocking of 47 bytes to socket 12
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] blocking send complete to socket 12
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4]-[[15051,1],8] tcp_peer_connected on socket 12
[cn5:13991] [[15051,0],4] routed_binomial_update: [[15051,1],8] --> [[15051,1],8]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]-[[15051,1],8] accepted: 192.168.1.5 - 192.168.1.5 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 267 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 645 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] tcp:set_module called for peer [[15051,1],8]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 738 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,1],8]
[cn5:13991] [[15051,0],4]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]:tcp:recv:handler allocate data region of size 61
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 195 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 419 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 190 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1217 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 275 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 104 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 199 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 126 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 102 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 579 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 290 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1251 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 159 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] connection_event_handler: working connection (13, 11) 192.168.1.5:56531
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4] accept_connection: 192.168.1.5:56531
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 932 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1141 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 118 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 577 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]:tcp:recv:handler called
[cn5:13991] [[15051,0],4] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 13
[cn5:13991] [[15051,0],4] waiting for connect ack from UNKNOWN
[cn5:13991] [[15051,0],4] connect ack received from UNKNOWN
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] connect-ack recvd from UNKNOWN
[cn5:13991] [[15051,0],4] mca_oob_tcp_recv_connect: connection from new peer
[cn5:13991] [[15051,0],4] connect-ack header from [[15051,1],9] is okay
[cn5:13991] [[15051,0],4] waiting for connect ack from [[15051,1],9]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] connect ack received from [[15051,1],9]
[cn5:13991] [[15051,0],4] connect-ack version from [[15051,1],9] matches ours
[cn5:13991] [[15051,0],4] connect-ack [[15051,1],9] authenticated
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:peer_accept called for peer [[15051,1],9] in state UNKNOWN on socket 13
[cn5:13991] [[15051,0],4] SEND CONNECT ACK
[cn5:13991] [[15051,0],4] send blocking of 47 bytes to socket 13
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 335 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] blocking send complete to socket 13
[cn5:13991] [[15051,0],4]-[[15051,1],9] tcp_peer_connected on socket 13
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 109 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] routed_binomial_update: [[15051,1],9] --> [[15051,1],9]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]-[[15051,1],9] accepted: 192.168.1.5 - 192.168.1.5 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1291 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4] tcp:set_module called for peer [[15051,1],9]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 810 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 661 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,1],9]
[cn5:13991] [[15051,0],4]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:tcp:recv:handler allocate data region of size 61
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 221 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1462 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 683 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 757 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 829 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] connection_event_handler: working connection (12, 0) 192.168.1.3:41971
[cn3:21242] [[15051,0],2] accept_connection: 192.168.1.3:41971
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] connection_event_handler: working connection (13, 0) 192.168.1.2:39186
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] accept_connection: 192.168.1.2:39186
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1110 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 936 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 655 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1]:tcp:recv:handler called
[cn2:00785] [[15051,0],1] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 13
[cn2:00785] [[15051,0],1] waiting for connect ack from UNKNOWN
[cn2:00785] [[15051,0],1] connect ack received from UNKNOWN
[cn2:00785] [[15051,0],1] connect-ack recvd from UNKNOWN
[cn2:00785] [[15051,0],1] mca_oob_tcp_recv_connect: connection from new peer
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] connect-ack header from [[15051,1],2] is okay
[cn2:00785] [[15051,0],1] waiting for connect ack from [[15051,1],2]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] connect ack received from [[15051,1],2]
[cn2:00785] [[15051,0],1] connect-ack version from [[15051,1],2] matches ours
[cn2:00785] [[15051,0],1] connect-ack [[15051,1],2] authenticated
[cn3:21242] [[15051,0],2]:tcp:recv:handler called
[cn3:21242] [[15051,0],2] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 12
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 160 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:peer_accept called for peer [[15051,1],2] in state UNKNOWN on socket 13
[cn2:00785] [[15051,0],1] SEND CONNECT ACK
[cn2:00785] [[15051,0],1] send blocking of 47 bytes to socket 13
[cn3:21242] [[15051,0],2] waiting for connect ack from UNKNOWN
[cn3:21242] [[15051,0],2] connect ack received from UNKNOWN
[cn3:21242] [[15051,0],2] connect-ack recvd from UNKNOWN
[cn3:21242] [[15051,0],2] mca_oob_tcp_recv_connect: connection from new peer
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] blocking send complete to socket 13
[cn3:21242] [[15051,0],2] connect-ack header from [[15051,1],4] is okay
[cn3:21242] [[15051,0],2] waiting for connect ack from [[15051,1],4]
[cn3:21242] [[15051,0],2] connect ack received from [[15051,1],4]
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn2:00785] [[15051,0],1]-[[15051,1],2] tcp_peer_connected on socket 13
[cn2:00785] [[15051,0],1] routed_binomial_update: [[15051,1],2] --> [[15051,1],2]
[cn3:21242] [[15051,0],2] connect-ack version from [[15051,1],4] matches ours
[cn3:21242] [[15051,0],2] connect-ack [[15051,1],4] authenticated
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]-[[15051,1],2] accepted: 192.168.1.2 - 192.168.1.2 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn3:21242] [[15051,0],2] tcp:peer_accept called for peer [[15051,1],4] in state UNKNOWN on socket 12
[cn3:21242] [[15051,0],2] SEND CONNECT ACK
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] send blocking of 47 bytes to socket 12
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] tcp:set_module called for peer [[15051,1],2]
[cn3:21242] [[15051,0],2] blocking send complete to socket 12
[cn3:21242] [[15051,0],2]-[[15051,1],4] tcp_peer_connected on socket 12
[cn3:21242] [[15051,0],2] routed_binomial_update: [[15051,1],4] --> [[15051,1],4]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,1],4] accepted: 192.168.1.3 - 192.168.1.3 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 575 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] tcp:set_module called for peer [[15051,1],4]
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 728 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 271 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,1],4]
[cn3:21242] [[15051,0],2]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:tcp:recv:handler allocate data region of size 61
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 810 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1217 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 500 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,1],2]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate data region of size 61
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1307 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 185 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 275 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 247 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 176 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 172 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 374 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 286 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 401 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 305 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 442 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 267 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 267 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 738 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 190 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 263 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 738 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 190 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 304 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 160 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 263 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 289 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 304 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 160 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 289 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 206 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 206 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 711 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 711 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 394 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 664 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 206 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1121 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 851 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] connection_event_handler: working connection (13, 11) 192.168.1.3:41972
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] accept_connection: 192.168.1.3:41972
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 934 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] connection_event_handler: working connection (14, 11) 192.168.1.2:39187
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] accept_connection: 192.168.1.2:39187
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 679 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]:tcp:recv:handler called
[cn3:21242] [[15051,0],2] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 13
[cn3:21242] [[15051,0],2] waiting for connect ack from UNKNOWN
[cn3:21242] [[15051,0],2] connect ack received from UNKNOWN
[cn3:21242] [[15051,0],2] connect-ack recvd from UNKNOWN
[cn3:21242] [[15051,0],2] mca_oob_tcp_recv_connect: connection from new peer
[cn3:21242] [[15051,0],2] connect-ack header from [[15051,1],5] is okay
[cn3:21242] [[15051,0],2] waiting for connect ack from [[15051,1],5]
[cn3:21242] [[15051,0],2] connect ack received from [[15051,1],5]
[cn3:21242] [[15051,0],2] connect-ack version from [[15051,1],5] matches ours
[cn3:21242] [[15051,0],2] connect-ack [[15051,1],5] authenticated
[cn3:21242] [[15051,0],2] tcp:peer_accept called for peer [[15051,1],5] in state UNKNOWN on socket 13
[cn3:21242] [[15051,0],2] SEND CONNECT ACK
[cn3:21242] [[15051,0],2] send blocking of 47 bytes to socket 13
[cn3:21242] [[15051,0],2] blocking send complete to socket 13
[cn3:21242] [[15051,0],2]-[[15051,1],5] tcp_peer_connected on socket 13
[cn3:21242] [[15051,0],2] routed_binomial_update: [[15051,1],5] --> [[15051,1],5]
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1]:tcp:recv:handler called
[cn2:00785] [[15051,0],1] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 14
[cn2:00785] [[15051,0],1] waiting for connect ack from UNKNOWN
[cn2:00785] [[15051,0],1] connect ack received from UNKNOWN
[cn2:00785] [[15051,0],1] connect-ack recvd from UNKNOWN
[cn2:00785] [[15051,0],1] mca_oob_tcp_recv_connect: connection from new peer
[cn2:00785] [[15051,0],1] connect-ack header from [[15051,1],3] is okay
[cn2:00785] [[15051,0],1] waiting for connect ack from [[15051,1],3]
[cn3:21242] [[15051,0],2]-[[15051,1],5] accepted: 192.168.1.3 - 192.168.1.3 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn2:00785] [[15051,0],1] connect ack received from [[15051,1],3]
[cn2:00785] [[15051,0],1] connect-ack version from [[15051,1],3] matches ours
[cn2:00785] [[15051,0],1] connect-ack [[15051,1],3] authenticated
[cn2:00785] [[15051,0],1] tcp:peer_accept called for peer [[15051,1],3] in state UNKNOWN on socket 14
[cn2:00785] [[15051,0],1] SEND CONNECT ACK
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] send blocking of 47 bytes to socket 14
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1167 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] blocking send complete to socket 14
[cn3:21242] [[15051,0],2] tcp:set_module called for peer [[15051,1],5]
[cn2:00785] [[15051,0],1]-[[15051,1],3] tcp_peer_connected on socket 14
[cn2:00785] [[15051,0],1] routed_binomial_update: [[15051,1],3] --> [[15051,1],3]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1]-[[15051,1],3] accepted: 192.168.1.2 - 192.168.1.2 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1006 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,1],5]
[cn3:21242] [[15051,0],2]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:tcp:recv:handler allocate data region of size 61
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:set_module called for peer [[15051,1],3]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 799 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 810 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,1],3]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate data region of size 61
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 693 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1462 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 276 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 532 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1280 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 108 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 448 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 360 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1358 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 176 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 374 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 104 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 190 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 286 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 267 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 738 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 190 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 105 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 419 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 121 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 336 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 945 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 551 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1155 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] connection_event_handler: working connection (13, 0) 192.168.1.4:42967
[cn4:17654] [[15051,0],3] accept_connection: 192.168.1.4:42967
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 908 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 678 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3]:tcp:recv:handler called
[cn4:17654] [[15051,0],3] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 13
[cn4:17654] [[15051,0],3] waiting for connect ack from UNKNOWN
[cn4:17654] [[15051,0],3] connect ack received from UNKNOWN
[cn4:17654] [[15051,0],3] connect-ack recvd from UNKNOWN
[cn4:17654] [[15051,0],3] mca_oob_tcp_recv_connect: connection from new peer
[cn4:17654] [[15051,0],3] connect-ack header from [[15051,1],6] is okay
[cn4:17654] [[15051,0],3] waiting for connect ack from [[15051,1],6]
[cn4:17654] [[15051,0],3] connect ack received from [[15051,1],6]
[cn4:17654] [[15051,0],3] connect-ack version from [[15051,1],6] matches ours
[cn4:17654] [[15051,0],3] connect-ack [[15051,1],6] authenticated
[cn4:17654] [[15051,0],3] tcp:peer_accept called for peer [[15051,1],6] in state UNKNOWN on socket 13
[cn4:17654] [[15051,0],3] SEND CONNECT ACK
[cn4:17654] [[15051,0],3] send blocking of 47 bytes to socket 13
[cn4:17654] [[15051,0],3] blocking send complete to socket 13
[cn4:17654] [[15051,0],3]-[[15051,1],6] tcp_peer_connected on socket 13
[cn4:17654] [[15051,0],3] routed_binomial_update: [[15051,1],6] --> [[15051,1],6]
[cn4:17654] [[15051,0],3]-[[15051,1],6] accepted: 192.168.1.4 - 192.168.1.4 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] tcp:set_module called for peer [[15051,1],6]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 260 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1217 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,1],6]
[cn4:17654] [[15051,0],3]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]:tcp:recv:handler allocate data region of size 61
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 275 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 369 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 481 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 108 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 160 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 289 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 434 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 926 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 472 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] connection_event_handler: working connection (14, 11) 192.168.1.4:42968
[cn4:17654] [[15051,0],3] accept_connection: 192.168.1.4:42968
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1452 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3]:tcp:recv:handler called
[cn4:17654] [[15051,0],3] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 14
[cn4:17654] [[15051,0],3] waiting for connect ack from UNKNOWN
[cn4:17654] [[15051,0],3] connect ack received from UNKNOWN
[cn4:17654] [[15051,0],3] connect-ack recvd from UNKNOWN
[cn4:17654] [[15051,0],3] mca_oob_tcp_recv_connect: connection from new peer
[cn4:17654] [[15051,0],3] connect-ack header from [[15051,1],7] is okay
[cn4:17654] [[15051,0],3] waiting for connect ack from [[15051,1],7]
[cn4:17654] [[15051,0],3] connect ack received from [[15051,1],7]
[cn4:17654] [[15051,0],3] connect-ack version from [[15051,1],7] matches ours
[cn4:17654] [[15051,0],3] connect-ack [[15051,1],7] authenticated
[cn4:17654] [[15051,0],3] tcp:peer_accept called for peer [[15051,1],7] in state UNKNOWN on socket 14
[cn4:17654] [[15051,0],3] SEND CONNECT ACK
[cn4:17654] [[15051,0],3] send blocking of 47 bytes to socket 14
[cn4:17654] [[15051,0],3] blocking send complete to socket 14
[cn4:17654] [[15051,0],3]-[[15051,1],7] tcp_peer_connected on socket 14
[cn4:17654] [[15051,0],3] routed_binomial_update: [[15051,1],7] --> [[15051,1],7]
[cn4:17654] [[15051,0],3]-[[15051,1],7] accepted: 192.168.1.4 - 192.168.1.4 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 709 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] tcp:set_module called for peer [[15051,1],7]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 810 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 103 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 925 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,1],7]
[cn4:17654] [[15051,0],3]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]:tcp:recv:handler allocate data region of size 61
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 494 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,1],8]
[cn5:13991] [[15051,0],4]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] RECVD COMPLETE MESSAGE FROM [[15051,1],8] (ORIGIN [[15051,1],8]) OF 61 BYTES FOR DEST [[15051,0],4] TAG 1
[cn5:13991] [[15051,0],4] DELIVERING TO RML
[cn5:13991] [[15051,0],4] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] message received 61 bytes from [[15051,1],8] for tag 1
[cn5:13991] [[15051,0],4] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn5:13991] [[15051,0],4] orted_recv: received sync+nidmap from local proc [[15051,1],8]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,1],8] at tag 20
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,1],8] at tag 20
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,1],8]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,1],8]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,1],8]:20
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,1],8]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,1],8]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,1],8]:20
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,1],8]) --> [[15051,1],8]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,1],8] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,1],8]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,1],8]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,1],8]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,1],8] OF 14140 BYTES ON SOCKET 12
[cn5:13991] [[15051,0],4]-[[15051,1],8] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,1],4]
[cn3:21242] [[15051,0],2]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] RECVD COMPLETE MESSAGE FROM [[15051,1],4] (ORIGIN [[15051,1],4]) OF 61 BYTES FOR DEST [[15051,0],2] TAG 1
[cn3:21242] [[15051,0],2] DELIVERING TO RML
[cn3:21242] [[15051,0],2] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] message received 61 bytes from [[15051,1],4] for tag 1
[cn3:21242] [[15051,0],2] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 273 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] orted_recv: received sync+nidmap from local proc [[15051,1],4]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 108 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,1],9]
[cn5:13991] [[15051,0],4]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] RECVD COMPLETE MESSAGE FROM [[15051,1],9] (ORIGIN [[15051,1],9]) OF 61 BYTES FOR DEST [[15051,0],4] TAG 1
[cn5:13991] [[15051,0],4] DELIVERING TO RML
[cn5:13991] [[15051,0],4] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] message received 61 bytes from [[15051,1],9] for tag 1
[cn5:13991] [[15051,0],4] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn5:13991] [[15051,0],4] orted_recv: received sync+nidmap from local proc [[15051,1],9]
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,1],2]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] RECVD COMPLETE MESSAGE FROM [[15051,1],2] (ORIGIN [[15051,1],2]) OF 61 BYTES FOR DEST [[15051,0],1] TAG 1
[cn2:00785] [[15051,0],1] DELIVERING TO RML
[cn2:00785] [[15051,0],1] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] message received 61 bytes from [[15051,1],2] for tag 1
[cn2:00785] [[15051,0],1] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn2:00785] [[15051,0],1] orted_recv: received sync+nidmap from local proc [[15051,1],2]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,1],4] at tag 20
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,1],4] at tag 20
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,1],4]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,1],4]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,1],4]:20
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,1],4]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,1],4]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,1],9] at tag 20
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,1],9] at tag 20
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,1],9]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,1],9]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,1],9]:20
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,1],9]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,1],9]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,1],9]:20
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,1],4]:20
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,1],4]) --> [[15051,1],4]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,1],4] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,1],4]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,1],4]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,1],4]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,1],4] OF 14140 BYTES ON SOCKET 12
[cn3:21242] [[15051,0],2]-[[15051,1],4] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,1],9]) --> [[15051,1],9]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,1],9] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,1],9]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,1],9]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,1],9]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,1],9] OF 14140 BYTES ON SOCKET 13
[cn5:13991] [[15051,0],4]-[[15051,1],9] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 5
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 5
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:5
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 351 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:5
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,1],2] at tag 20
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,1],2] at tag 20
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,1],2]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,1],2]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,1],2]:20
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,1],2]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 179 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 103 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,1],2]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,1],2]:20
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,1],2]) --> [[15051,1],2]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,1],2] - queueing for send
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,1],2]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,1],2]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,1],2]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,1],2] OF 14140 BYTES ON SOCKET 13
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1]-[[15051,1],2] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 83 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 95 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 273 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 82 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 108 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 108 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,1],5]
[cn3:21242] [[15051,0],2]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] RECVD COMPLETE MESSAGE FROM [[15051,1],5] (ORIGIN [[15051,1],5]) OF 61 BYTES FOR DEST [[15051,0],2] TAG 1
[cn3:21242] [[15051,0],2] DELIVERING TO RML
[cn3:21242] [[15051,0],2] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] message received 61 bytes from [[15051,1],5] for tag 1
[cn3:21242] [[15051,0],2] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn3:21242] [[15051,0],2] orted_recv: received sync+nidmap from local proc [[15051,1],5]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,1],6]
[cn4:17654] [[15051,0],3]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] RECVD COMPLETE MESSAGE FROM [[15051,1],6] (ORIGIN [[15051,1],6]) OF 61 BYTES FOR DEST [[15051,0],3] TAG 1
[cn4:17654] [[15051,0],3] DELIVERING TO RML
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,1],3]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] RECVD COMPLETE MESSAGE FROM [[15051,1],3] (ORIGIN [[15051,1],3]) OF 61 BYTES FOR DEST [[15051,0],1] TAG 1
[cn2:00785] [[15051,0],1] DELIVERING TO RML
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,1],5] at tag 20
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,1],5] at tag 20
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,1],5]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,1],5]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,1],5]:20
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,1],5]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] message received 61 bytes from [[15051,1],6] for tag 1
[cn2:00785] [[15051,0],1] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] message received 61 bytes from [[15051,1],3] for tag 1
[cn2:00785] [[15051,0],1] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,1],5]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,1],5]:20
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,1],5]) --> [[15051,1],5]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,1],5] - queueing for send
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 156 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn4:17654] [[15051,0],3] orted_recv: received sync+nidmap from local proc [[15051,1],6]
[cn2:00785] [[15051,0],1] orted_recv: received sync+nidmap from local proc [[15051,1],3]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,1],5]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,1],5]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,1],5]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,1],6] at tag 20
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,1],6] at tag 20
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,1],6]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,1],3] at tag 20
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,1],3] at tag 20
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,1],3]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,1],3]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,1],3]:20
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,1],5] OF 14140 BYTES ON SOCKET 13
[cn3:21242] [[15051,0],2]-[[15051,1],5] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,1],6]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,1],6]:20
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,1],6]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,1],6]
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,1],3]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,1],3]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,1],3]:20
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 5
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,1],6]:20
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,1],6]) --> [[15051,1],6]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,1],6] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,1],6]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,1],6]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,1],3]) --> [[15051,1],3]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,1],3] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,1],3]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,1],3]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 5
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,1],6]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,1],3]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,1],3] OF 14140 BYTES ON SOCKET 14
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 429 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,1],6] OF 14140 BYTES ON SOCKET 13
[cn2:00785] [[15051,0],1]-[[15051,1],3] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:5
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3]-[[15051,1],6] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 5
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 5
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:5
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 83 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:5
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:5
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 179 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 103 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 179 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 273 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 482 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 273 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 108 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 156 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 83 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 351 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 108 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 429 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,1],7]
[cn4:17654] [[15051,0],3]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] RECVD COMPLETE MESSAGE FROM [[15051,1],7] (ORIGIN [[15051,1],7]) OF 61 BYTES FOR DEST [[15051,0],3] TAG 1
[cn4:17654] [[15051,0],3] DELIVERING TO RML
[cn4:17654] [[15051,0],3] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] message received 61 bytes from [[15051,1],7] for tag 1
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn4:17654] [[15051,0],3] orted_recv: received sync+nidmap from local proc [[15051,1],7]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,1],7] at tag 20
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,1],7] at tag 20
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,1],7]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,1],7]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,1],7]:20
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,1],7]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,1],7]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,1],7]:20
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,1],7]) --> [[15051,1],7]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,1],7] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,1],7]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,1],7]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,1],7]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 83 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,1],7] OF 14140 BYTES ON SOCKET 14
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 555 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3]-[[15051,1],7] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 5
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 5
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 103 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:5
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 83 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:5
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1231 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 943 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 179 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4001 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 351 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1969 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 555 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1937 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 83 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 3034 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 555 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 3034 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 83 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1849 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 2348 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 3081 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 2531 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1814 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 2352 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 3304 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 563 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1880 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1880 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1555 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 301 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 464 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 480 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 464 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] mca_oob_tcp_listen_thread: new connection: (20, 11) 192.168.1.1:59869
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 231 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 555 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 343 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 175 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 83 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 104 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 193 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 120 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] connection_handler: working connection (20, 0) 192.168.1.1:59869
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] accept_connection: 192.168.1.1:59869
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 102 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 482 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 5
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 5
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1880 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 5
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1880 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 120 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 83 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 5
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],4] for tag 5
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] plm:base:receive processing msg
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0] plm:base:receive update proc state command from [[15051,0],4]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for job [15051,1]
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 8 state RUNNING exit_code 0
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 9 state RUNNING exit_code 0
[cn1:13986] [[15051,0],0] plm:base:receive done processing commands
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],1] for tag 5
[cn1:13986] [[15051,0],0] plm:base:receive processing msg
[cn1:13986] [[15051,0],0] plm:base:receive update proc state command from [[15051,0],1]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for job [15051,1]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 2 state RUNNING exit_code 0
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 3 state RUNNING exit_code 0
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] plm:base:receive done processing commands
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],3] for tag 5
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] plm:base:receive processing msg
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] plm:base:receive update proc state command from [[15051,0],3]
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for job [15051,1]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 6 state RUNNING exit_code 0
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 7 state RUNNING exit_code 0
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 225 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] plm:base:receive done processing commands
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],2] for tag 5
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] plm:base:receive processing msg
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] plm:base:receive update proc state command from [[15051,0],2]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for job [15051,1]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 4 state RUNNING exit_code 0
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 5 state RUNNING exit_code 0
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] plm:base:receive done processing commands
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 464 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 311
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 118 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 311 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 464 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 311
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 311 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 81
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 81 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called
[cn1:13986] [[15051,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 20
[cn1:13986] [[15051,0],0] waiting for connect ack from UNKNOWN
[cn1:13986] [[15051,0],0] connect ack received from UNKNOWN
[cn1:13986] [[15051,0],0] connect-ack recvd from UNKNOWN
[cn1:13986] [[15051,0],0] mca_oob_tcp_recv_connect: connection from new peer
[cn1:13986] [[15051,0],0] connect-ack header from [[15051,1],0] is okay
[cn1:13986] [[15051,0],0] waiting for connect ack from [[15051,1],0]
[cn1:13986] [[15051,0],0] connect ack received from [[15051,1],0]
[cn1:13986] [[15051,0],0] connect-ack version from [[15051,1],0] matches ours
[cn1:13986] [[15051,0],0] connect-ack [[15051,1],0] authenticated
[cn1:13986] [[15051,0],0] tcp:peer_accept called for peer [[15051,1],0] in state UNKNOWN on socket 20
[cn1:13986] [[15051,0],0] SEND CONNECT ACK
[cn1:13986] [[15051,0],0] send blocking of 47 bytes to socket 20
[cn1:13986] [[15051,0],0] blocking send complete to socket 20
[cn1:13986] [[15051,0],0]-[[15051,1],0] tcp_peer_connected on socket 20
[cn1:13986] [[15051,0],0] routed_binomial_update: [[15051,1],0] --> [[15051,1],0]
[cn1:13986] [[15051,0],0]-[[15051,1],0] accepted: 192.168.1.1 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:14001] progress: initialized event flag to: 3
[cn1:14001] progress: initialized yield_when_idle to: false
[cn1:14001] progress: initialized num users to: 0
[cn1:14001] progress: initialized poll rate to: 23940000
[cn1:14001] progress: event_users_increment setting count to 1
[cn1:14001] mca: base: components_register: registering ess components
[cn1:14001] mca: base: components_register: found loaded component env
[cn1:14001] mca: base: components_register: component env has no register or open function
[cn1:14001] mca: base: components_open: opening ess components
[cn1:14001] mca: base: components_open: found loaded component env
[cn1:14001] mca: base: components_open: component env open function successful
[cn1:14001] mca:base:select: Auto-selecting ess components
[cn1:14001] mca:base:select:(  ess) Querying component [env]
[cn1:14001] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn1:14001] mca:base:select:(  ess) Selected component [env]
[cn1:14001] ess:env set name to [[15051,1],0]
[cn1:14001] mca: base: components_register: registering oob components
[cn1:14001] mca: base: components_register: found loaded component tcp
[cn1:14001] mca: base: components_register: component tcp register function successful
[cn1:14001] mca: base: components_open: opening oob components
[cn1:14001] mca: base: components_open: found loaded component tcp
[cn1:14001] mca: base: components_open: component tcp open function successful
[cn1:14001] mca:oob:select: checking available component tcp
[cn1:14001] mca:oob:select: Querying component [tcp]
[cn1:14001] oob:tcp: component_available called
[cn1:14001] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn1:14001] [[15051,1],0] oob:tcp:init rejecting loopback interface lo
[cn1:14001] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn1:14001] [[15051,1],0] oob:tcp:init creating module for V4 address on interface eth0
[cn1:14001] [[15051,1],0] creating OOB-TCP module for interface eth0
[cn1:14001] [[15051,1],0] oob:tcp:init adding 192.168.1.1 to our list of V4 connections
[cn1:14001] WORKING INTERFACE 3 KERNEL INDEX 3 FAMILY: V4
[cn1:14001] [[15051,1],0] oob:tcp:init creating module for V4 address on interface eth1
[cn1:14001] [[15051,1],0] creating OOB-TCP module for interface eth1
[cn1:14001] [[15051,1],0] oob:tcp:init adding 192.168.10.2 to our list of V4 connections
[cn1:14001] [[15051,1],0] TCP STARTUP
[cn1:14001] [[15051,1],0] attempting to bind to IPv4 port 0
[cn1:14001] [[15051,1],0] assigned IPv4 port 38811
[cn1:14001] mca:oob:select: Adding component to end
[cn1:14001] mca:oob:select: Found 1 active transports
[cn1:14001] mca: base: components_register: registering rml components
[cn1:14001] mca: base: components_register: found loaded component oob
[cn1:14001] mca: base: components_register: component oob has no register or open function
[cn1:14001] mca: base: components_open: opening rml components
[cn1:14001] mca: base: components_open: found loaded component oob
[cn1:14001] mca: base: components_open: component oob open function successful
[cn1:14001] orte_rml_base_select: initializing rml component oob
[cn1:14001] mca: base: components_register: registering routed components
[cn1:14001] mca: base: components_register: found loaded component binomial
[cn1:14001] mca: base: components_register: component binomial has no register or open function
[cn1:14001] mca: base: components_register: found loaded component debruijn
[cn1:14001] mca: base: components_register: component debruijn has no register or open function
[cn1:14001] mca: base: components_register: found loaded component direct
[cn1:14001] mca: base: components_register: component direct has no register or open function
[cn1:14001] mca: base: components_register: found loaded component radix
[cn1:14001] mca: base: components_register: component radix register function successful
[cn1:14001] mca: base: components_open: opening routed components
[cn1:14001] mca: base: components_open: found loaded component binomial
[cn1:14001] mca: base: components_open: found loaded component debruijn
[cn1:14001[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 311
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 311 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 311 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 311 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 81 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] tcp:set_module called for peer [[15051,1],0]
[cn1:13986] [[15051,0],0] message received 311 bytes from [[15051,0],4] for tag 2
] mca: base: components_open: found loaded component direct
[cn1:14001] mca: base: components_open: found loaded component radix
[cn1:14001] mca:base:select: Auto-selecting routed components
[cn1:14001] mca:base:select:(routed) Querying component [binomial]
[cn1:14001] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn1:14001] mca:base:select:(routed) Querying component [debruijn]
[cn1:14001] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn1:14001] mca:base:select:(routed) Querying component [direct]
[cn1:14001] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn1:14001] mca:base:select:(routed) Querying component [radix]
[cn1:14001] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn1:14001] mca:base:select:(routed) Selected component [binomial]
[cn1:14001] mca: base: close: unloading component debruijn
[cn1:14001] mca: base: close: unloading component direct
[cn1:14001] mca: base: close: unloading component radix
[cn1:14001] orte_routed_base_select: initializing selected component binomial
[cn1:14001] mca: base: components_register: registering dstore components
[cn1:14001] mca: base: components_register: found loaded component hash
[cn1:14001] mca: base: components_register: component hash register function successful
[cn1:14001] mca: base: components_open: opening dstore components
[cn1:14001] mca: base: components_open: found loaded component hash
[cn1:14001] mca:dstore:select: checking available component hash
[cn1:14001] mca:dstore:select: Querying component [hash]
[cn1:14001] mca: base: components_register: registering grpcomm components
[cn1:14001] mca: base: components_register: found loaded component bad
[cn1:14001] mca: base: components_register: component bad has no register or open function
[cn1:14001] mca: base: components_open: opening grpcomm components
[cn1:14001] mca: base: components_open: found loaded component bad
[cn1:14001] mca: base: components_open: component bad open function successful
[cn1:14001] mca:base:select: Auto-selecting grpcomm components
[cn1:14001] mca:base:select:(grpcomm) Querying component [bad]
[cn1:14001] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn1:14001] mca:base:select:(grpcomm) Selected component [bad]
[cn1:14001] [[15051,1],0] grpcomm:base:receive start comm
[cn1:14001] [[15051,1],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn1:14001] [[15051,1],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn1:14001] [[15051,1],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn1:14001] [[15051,1],0] setting up session dir with
	tmpdir: /tmp
	host cn1
[cn1:14001] [[15051,1],0] posting recv
[cn1:14001] [[15051,1],0] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn1:14001] [[15051,1],0] posting recv
[cn1:14001] [[15051,1],0] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn1:14001] [[15051,1],0] posting recv
[cn1:14001] [[15051,1],0] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn1:14001] procdir: /tmp/openmpi-sessions-artpol@cn1_0/15051/1/0
[cn1:14001] jobdir: /tmp/openmpi-sessions-artpol@cn1_0/15051/1
[cn1:14001] top: openmpi-sessions-artpol@cn1_0
[cn1:14001] tmp: /tmp
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 986382337
[cn1:14001] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 986382337
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 986382337
[cn1:14001] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 986382337
[cn1:14001] [[15051,1],0] routed_binomial: init routes for proc job [15051,1]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
	daemon uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn1:14001] [[15051,1],0] [[15051,0],0] with daemon registering
[cn1:14001] [[15051,1],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn1:14001] [[15051,1],0] rml_send_buffer to peer [[15051,0[cn2:00788] progress: initialized event flag to: 3
[cn2:00788] progress: initialized yield_when_idle to: false
[cn2:00788] progress: initialized num users to: 0
[cn2:00788] progress: initialized poll rate to: 23940000
[cn2:00788] progress: event_users_increment setting count to 1
[cn4:17657] progress: initialized event flag to: 3
[cn4:17657] progress: initialized yield_when_idle to: false
[cn4:17657] progress: initialized num users to: 0
[cn4:17657] progress: initialized poll rate to: 23940000
[cn4:17657] progress: event_users_increment setting count to 1
[cn3:21245] progress: initialized event flag to: 3
[cn5:13994] progress: initialized event flag to: 3
[cn5:13994] progress: initialized yield_when_idle to: false
[cn5:13994] progress: initialized num users to: 0
[cn5:13994] progress: initialized poll rate to: 23940000
[cn5:13994] progress: event_users_increment setting count to 1
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 81
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 81 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 311
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 311 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 311
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 311 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 90
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 90 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 81 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 311 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 311 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 90 bytes from [[15051,0],2] for tag 2
],0] at tag 1
[cn1:14001] [[15051,1],0] registering sync waiting for ack
[cn1:14001] [[15051,1],0] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn1:14001] [[15051,1],0]: set_addr to uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn1:14001] [[15051,1],0]:set_addr checking if peer [[15051,0],0] is reachable via component tcp
[cn1:14001] [[15051,1],0] oob:tcp: working peer [[15051,0],0] address tcp://192.168.1.1,192.168.10.2:37478
[cn1:14001] [[15051,1],0] PEER [[15051,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:14001] [[15051,1],0] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:14001] [[15051,1],0]:tcp set addr for peer [[15051,0],0]
[cn1:14001] [[15051,1],0] PEER [[15051,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 3 INTERFACE eth1
[cn1:14001] [[15051,1],0] PASSING ADDR 192.168.10.2 TO INTERFACE eth1 AT KERNEL INDEX 3
[cn1:14001] [[15051,1],0]:tcp set addr for peer [[15051,0],0]
[cn1:14001] [[15051,1],0]: peer [[15051,0],0] is reachable via component tcp
[cn1:14001] [[15051,1],0] posting recv
[cn1:14001] [[15051,1],0] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn1:14001] [[15051,1],0] rml_send_msg to peer [[15051,0],0] at tag 1
[cn1:14001] [[15051,1],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14001] [[15051,1],0]:tcp:processing set_peer cmd for interface eth0
[cn1:14001] [[15051,1],0] SET_PEER ADDING PEER [[15051,0],0]
[cn1:14001] [[15051,1],0] set_peer: peer [[15051,0],0] is listening on interface eth0 net 192.168.1.1 port 37478
[cn1:14001] [[15051,1],0]:tcp:processing set_peer cmd for interface eth1
[cn1:14001] [[15051,1],0] SET_PEER ADDING PEER [[15051,0],0]
[cn1:14001] [[15051,1],0] set_peer: peer [[15051,0],0] is listening on interface eth1 net 192.168.10.2 port 37478
[cn1:14001] [[15051,1],0] oob:base:send to target [[15051,0],0]
[cn1:14001] [[15051,1],0] oob:tcp:send_nb to peer [[15051,0],0]:1
[cn1:14001] [[15051,1],0] tcp:send_nb to peer [[15051,0],0]
[cn1:14001] [[15051,1],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:14001] [[15051,1],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:1
[cn1:14001] [[15051,1],0] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:14001] [[15051,1],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],0]
[cn1:14001] [[15051,1],0] tcp:send_nb: initiating connection to [[15051,0],0]
[cn1:14001] [[15051,1],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],0]
[cn1:14001] [[15051,1],0] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] via interface eth0
[cn1:14001] [[15051,1],0] oob:tcp:peer creating socket to [[15051,0],0]
[cn1:14001] [[15051,1],0] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] via interface eth0 on socket 11
[cn1:14001] [[15051,1],0] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] on 192.168.1.1:37478 - 0 retries
[cn1:14001] [[15051,1],0] waiting for connect completion to [[15051,0],0] - activating send event
[cn1:14001] [[15051,1],0] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:14001] [[15051,1],0] tcp:send_handler CONNECTING
[cn1:14001] [[15051,1],0]:tcp:complete_connect called for peer [[15051,0],0] on socket 11
[cn1:14001] [[15051,1],0] tcp_peer_complete_connect: sending ack to [[15051,0],0]
[cn1:14001] [[15051,1],0] SEND CONNECT ACK
[cn1:14001] fetching data from INTERNAL dstore
[cn1:14001] dstore:hash:fetch: searching for key opal.cred on proc 986382337
[cn1:14001] dstore_hash:fetch key opal.cred for proc 986382337 not found
[cn1:14001] [[15051,1],0] send blocking of 47 bytes to socket 11
[cn1:14001] [[15051,1],0] blocking send complete to socket 11
[cn1:14001] [[15051,1],0] tcp_peer_complete_connect: [cn5:13995] progress: initialized event flag to: 3
[cn2:00789] progress: initialized event flag to: 3
[cn2:00789] progress: initialized yield_when_idle to: false
[cn2:00789] progress: initialized num users to: 0
[cn2:00789] progress: initialized poll rate to: 23940000
[cn2:00789] progress: event_users_increment setting count to 1
[cn4:17658] progress: initialized event flag to: 3
[cn4:17658] progress: initialized yield_when_idle to: false
[cn4:17658] progress: initialized num users to: 0
[cn4:17658] progress: initialized poll rate to: 23940000
[cn4:17658] progress: event_users_increment setting count to 1
[cn3:21245] progress: initialized yield_when_idle to: false
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 260
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 260 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 263
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 263 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 263
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 263 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 80
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 80 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 260 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 263 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 263 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 80 bytes from [[15051,0],2] for tag 2
setting read event on connection to [[15051,0],0]
[cn5:13995] progress: initialized yield_when_idle to: false
[cn5:13995] progress: initialized num users to: 0
[cn5:13995] progress: initialized poll rate to: 23940000
[cn5:13995] progress: event_users_increment setting count to 1
[cn2:00788] mca: base: components_register: registering ess components
[cn2:00788] mca: base: components_register: found loaded component env
[cn2:00788] mca: base: components_register: component env has no register or open function
[cn4:17657] mca: base: components_register: registering ess components
[cn4:17657] mca: base: components_register: found loaded component env
[cn4:17657] mca: base: components_register: component env has no register or open function
[cn3:21245] progress: initialized num users to: 0
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 500
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 500 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 500
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 500 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 87
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 87 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 500 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 500 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 87 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: registering ess components
[cn2:00788] mca: base: components_open: opening ess components
[cn2:00788] mca: base: components_open: found loaded component env
[cn2:00788] mca: base: components_open: component env open function successful
[cn2:00788] mca:base:select: Auto-selecting ess components
[cn2:00788] mca:base:select:(  ess) Querying component [env]
[cn2:00788] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn2:00788] mca:base:select:(  ess) Selected component [env]
[cn4:17657] mca: base: components_open: opening ess components
[cn4:17657] mca: base: components_open: found loaded component env
[cn4:17657] mca: base: components_open: component env open function successful
[cn4:17657] mca:base:select: Auto-selecting ess components
[cn4:17657] mca:base:select:(  ess) Querying component [env]
[cn4:17657] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn4:17657] mca:base:select:(  ess) Selected component [env]
[cn3:21245] progress: initialized poll rate to: 23940000
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 76
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 76 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 76
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 76 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 93
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 93 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 76 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 76 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 93 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: found loaded component env
[cn2:00788] ess:env set name to [[15051,1],2]
[cn4:17657] ess:env set name to [[15051,1],6]
[cn3:21245] progress: event_users_increment setting count to 1
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 121
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 121 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 592
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 592 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 81
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 81 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 121 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 592 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 81 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 260
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 260 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] mca: base: components_register: component env has no register or open function
[cn2:00789] mca: base: components_register: registering ess components
[cn2:00789] mca: base: components_register: found loaded component env
[cn2:00789] mca: base: components_register: component env has no register or open function
[cn2:00789] mca: base: components_open: opening ess components
[cn2:00789] mca: base: components_open: found loaded component env
[cn2:00789] mca: base: components_open: component env open function successful
[cn2:00789] mca:base:select: Auto-selecting ess components
[cn2:00789] mca:base:select:(  ess) Querying component [env]
[cn4:17658] mca: base: components_register: registering ess components
[cn3:21246] progress: initialized event flag to: 3
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 93
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 93 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 217
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 217 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 192
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 192 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 260 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] message received 93 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 217 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 192 bytes from [[15051,0],3] for tag 2
[cn3:21246] progress: initialized yield_when_idle to: false
[cn3:21246] progress: initialized num users to: 0
[cn3:21246] progress: initialized poll rate to: 23940000
[cn3:21246] progress: event_users_increment setting count to 1
[cn5:13994] mca: base: components_open: opening ess components
[cn2:00789] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn2:00789] mca:base:select:(  ess) Selected component [env]
[cn2:00789] ess:env set name to [[15051,1],3]
[cn4:17658] mca: base: components_register: found loaded component env
[cn4:17658] mca: base: components_register: component env has no register or open function
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 97
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 97 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 172
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 172 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 546
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 546 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 97 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 172 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 546 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_open: found loaded component env
[cn2:00788] mca: base: components_register: registering oob components
[cn2:00788] mca: base: components_register: found loaded component tcp
[cn4:17658] mca: base: components_open: opening ess components
[cn4:17658] mca: base: components_open: found loaded component env
[cn4:17658] mca: base: components_open: component env open function successful
[cn4:17658] mca:base:select: Auto-selecting ess components
[cn4:17658] mca:base:select:(  ess) Querying component [env]
[cn4:17658] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn4:17658] mca:base:select:(  ess) Selected component [env]
[cn4:17658] ess:env set name to [[15051,1],7]
[cn3:21245] mca: base: components_register: registering ess components
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 109
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 109 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 109 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 291
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 291 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 353
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 353 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 121
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 2256 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 121 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn5:13994] mca: base: components_open: component env open function successful
[cn2:00788] mca: base: components_register: component tcp register function successful
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17657] mca: base: components_register: registering oob components
[cn3:21245] mca: base: components_register: found loaded component env
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 291 bytes from [[15051,0],4] for tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] message received 353 bytes from [[15051,0],1] for tag 2
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 121 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 76
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 76 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1881 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 78
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 78 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 100 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 464 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 2199 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 93
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 93 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 464 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13994] mca:base:select: Auto-selecting ess components
[cn5:13994] mca:base:select:(  ess) Querying component [env]
[cn5:13994] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn5:13994] mca:base:select:(  ess) Selected component [env]
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn2:00788] mca: base: components_open: opening oob components
[cn2:00788] mca: base: components_open: found loaded component tcp
[cn2:00788] mca: base: components_open: component tcp open function successful
[cn2:00788] mca:oob:select: checking available component tcp
[cn2:00788] mca:oob:select: Querying component [tcp]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17657] mca: base: components_register: found loaded component tcp
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21245] mca: base: components_register: component env has no register or open function
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 76 bytes from [[15051,0],4] for tag 2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] message received 78 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 93 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 500
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 500 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 160
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 160 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 437
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 437 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,1],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 74
[cn5:13994] ess:env set name to [[15051,1],8]
[cn2:00788] oob:tcp: component_available called
[cn4:17657] mca: base: components_register: component tcp register function successful
[cn3:21245] mca: base: components_open: opening ess components
[cn1:14001] [[15051,1],0]:tcp:recv:handler called for peer [[15051,0],0]
[cn1:14001] [[15051,1],0] RECV CONNECT ACK FROM [[15051,0],0] ON SOCKET 11
[cn1:14001] [[15051,1],0] waiting for connect ack from [[15051,0],0]
[cn1:14001] [[15051,1],0] connect ack received from [[15051,0],0]
[cn1:14001] [[15051,1],0] connect-ack recvd from [[15051,0],0]
[cn1:14001] [[15051,1],0] connect-ack header from [[15051,0],0] is okay
[cn1:14001] [[15051,1],0] waiting for connect ack from [[15051,0],0]
[cn1:14001] [[15051,1],0] connect ack received from [[15051,0],0]
[cn1:14001] [[15051,1],0] connect-ack version from [[15051,0],0] matches ours
[cn1:14001] [[15051,1],0] connect-ack [[15051,0],0] authenticated
[cn1:14001] [[15051,1],0]-[[15051,0],0] tcp_peer_connected on socket 11
[cn1:14001] [[15051,1],0]-[[15051,0],0] connected: 192.168.1.1 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:14001] [[15051,1],0]:tcp:recv:handler starting send/recv events
[cn1:14001] [[15051,1],0] tcp:set_module called for peer [[15051,0],0]
[cn1:14001] [[15051,1],0] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:14001] [[15051,1],0] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:14001] [[15051,1],0] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 74 BYTES ON SOCKET 11
[cn1:14001] [[15051,1],0]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 500 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 160 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 437 bytes from [[15051,0],2] for tag 2
[cn5:13995] mca: base: components_register: registering ess components
[cn2:00788] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn2:00788] [[15051,1],2] oob:tcp:init rejecting loopback interface lo
[cn2:00788] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn2:00788] [[15051,1],2] oob:tcp:init creating module for V4 address on interface eth0
[cn2:00788] [[15051,1],2] creating OOB-TCP module for interface eth0
[cn2:00788] [[15051,1],2] oob:tcp:init adding 192.168.1.2 to our list of V4 connections
[cn2:00788] [[15051,1],2] TCP STARTUP
[cn4:17657] mca: base: components_open: opening oob components
[cn4:17657] mca: base: components_open: found loaded component tcp
[cn3:21245] mca: base: components_open: found loaded component env
[cn3:21245] mca: base: components_open: component env open function successful
[cn3:21245] mca:base:select: Auto-selecting ess components
[cn3:21245] mca:base:select:(  ess) Querying component [env]
[cn3:21245] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn3:21245] mca:base:select:(  ess) Selected component [env]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 247
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 247 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 271
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 271 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 76
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 76 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 247 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 271 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 76 bytes from [[15051,0],2] for tag 2
[cn5:13995] mca: base: components_register: found loaded component env
[cn2:00788] [[15051,1],2] attempting to bind to IPv4 port 0
[cn2:00788] [[15051,1],2] assigned IPv4 port 37167
[cn2:00788] mca:oob:select: Adding component to end
[cn2:00788] mca:oob:select: Found 1 active transports
[cn4:17657] mca: base: components_open: component tcp open function successful
[cn4:17657] mca:oob:select: checking available component tcp
[cn4:17657] mca:oob:select: Querying component [tcp]
[cn4:17657] oob:tcp: component_available called
[cn3:21245] ess:env set name to [[15051,1],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 121
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 121 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 500
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 500 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 121 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 500 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 192
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 192 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_register: component env has no register or open function
[cn2:00789] mca: base: components_register: registering oob components
[cn4:17657] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn4:17657] [[15051,1],6] oob:tcp:init rejecting loopback interface lo
[cn4:17657] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn4:17657] [[15051,1],6] oob:tcp:init creating module for V4 address on interface eth0
[cn4:17657] [[15051,1],6] creating OOB-TCP module for interface eth0
[cn4:17657] [[15051,1],6] oob:tcp:init adding 192.168.1.4 to our list of V4 connections
[cn4:17657] [[15051,1],6] TCP STARTUP
[cn3:21246] mca: base: components_register: registering ess components
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 546
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 546 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 247
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 247 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 192 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] message received 546 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 247 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 172
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 172 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 546
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 546 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21246] mca: base: components_register: found loaded component env
[cn3:21246] mca: base: components_register: component env has no register or open function
[cn5:13995] mca: base: components_open: opening ess components
[cn5:13995] mca: base: components_open: found loaded component env
[cn5:13995] mca: base: components_open: component env open function successful
[cn5:13995] mca:base:select: Auto-selecting ess components
[cn5:13995] mca:base:select:(  ess) Querying component [env]
[cn5:13995] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn5:13995] mca:base:select:(  ess) Selected component [env]
[cn5:13995] ess:env set name to [[15051,1],9]
[cn2:00789] mca: base: components_register: found loaded component tcp
[cn4:17657] [[15051,1],6] attempting to bind to IPv4 port 0
[cn4:17657] [[15051,1],6] assigned IPv4 port 50185
[cn4:17657] mca:oob:select: Adding component to end
[cn4:17657] mca:oob:select: Found 1 active transports
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 172 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 546 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: registering oob components
[cn2:00789] mca: base: components_register: component tcp register function successful
[cn4:17658] mca: base: components_register: registering oob components
[cn4:17658] mca: base: components_register: found loaded component tcp
[cn3:21246] mca: base: components_open: opening ess components
[cn3:21246] mca: base: components_open: found loaded component env
[cn3:21246] mca: base: components_open: component env open function successful
[cn3:21246] mca:base:select: Auto-selecting ess components
[cn3:21246] mca:base:select:(  ess) Querying component [env]
[cn3:21246] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn3:21246] mca:base:select:(  ess) Selected component [env]
[cn3:21246] ess:env set name to [[15051,1],5]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 401
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 401 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 172
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 172 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 401 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 172 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: found loaded component tcp
[cn2:00789] mca: base: components_open: opening oob components
[cn2:00789] mca: base: components_open: found loaded component tcp
[cn2:00789] mca: base: components_open: component tcp open function successful
[cn2:00789] mca:oob:select: checking available component tcp
[cn2:00789] mca:oob:select: Querying component [tcp]
[cn2:00789] oob:tcp: component_available called
[cn4:17658] mca: base: components_register: component tcp register function successful
[cn3:21245] mca: base: components_register: registering oob components
[cn3:21245] mca: base: components_register: found loaded component tcp
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 88
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 88 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 401
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 401 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 88 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 401 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 247
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 247 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 263
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 263 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 2016 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 305
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 305 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 401
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 116 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 401 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 2016 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13995] mca: base: components_register: registering oob components
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00789] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn4:17658] mca: base: components_open: opening oob components
[cn4:17658] mca: base: components_open: found loaded component tcp
[cn4:17658] mca: base: components_open: component tcp open function successful
[cn4:17658] mca:oob:select: checking available component tcp
[cn4:17658] mca:oob:select: Querying component [tcp]
[cn4:17658] oob:tcp: component_available called
[cn3:21245] mca: base: components_register: component tcp register function successful
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:14002] progress: initialized event flag to: 3
[cn1:14002] progress: initialized yield_when_idle to: false
[cn1:14002] progress: initialized num users to: 0
[cn1:14002] progress: initialized poll rate to: 23940000
[cn1:14002] progress: event_users_increment setting count to 1
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 247 bytes from [[15051,0],4] for tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 263 bytes from [[15051,0],1] for tag 2
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 305 bytes from [[15051,0],3] for tag 2
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1800 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] message received 401 bytes from [[15051,0],2] for tag 2
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn5:13994] mca: base: components_register: component tcp register function successful
[cn5:13994] mca: base: components_open: opening oob components
[cn5:13994] mca: base: components_open: found loaded component tcp
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00788] mca: base: components_register: registering rml components
[cn2:00788] mca: base: components_register: found loaded component oob
[cn2:00788] mca: base: components_register: component oob has no register or open function
[cn4:17658] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn4:17658] [[15051,1],7] oob:tcp:init rejecting loopback interface lo
[cn4:17658] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn4:17658] [[15051,1],7] oob:tcp:init creating module for V4 address on interface eth0
[cn3:21245] mca: base: components_open: opening oob components
[cn3:21245] mca: base: components_open: found loaded component tcp
[cn3:21245] mca: base: components_open: component tcp open function successful
[cn3:21245] mca:oob:select: checking available component tcp
[cn3:21245] mca:oob:select: Querying component [tcp]
[cn3:21245] oob:tcp: component_available called
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 271
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 271 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 304
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 304 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 442
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 442 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 88
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 88 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 271 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn1:13986] [[15051,0],0] message received 304 bytes from [[15051,0],1] for tag 2
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 442 bytes from [[15051,0],3] for tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] message received 88 bytes from [[15051,0],2] for tag 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 326 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 159
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 159 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 2016 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 2560 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 247
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 247 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 263
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 263 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] mca: base: components_open: component tcp open function successful
[cn5:13994] mca:oob:select: checking available component tcp
[cn5:13994] mca:oob:select: Querying component [tcp]
[cn5:13994] oob:tcp: component_available called
[cn2:00788] mca: base: components_open: opening rml components
[cn2:00788] mca: base: components_open: found loaded component oob
[cn2:00788] mca: base: components_open: component oob open function successful
[cn2:00788] orte_rml_base_select: initializing rml component oob
[cn4:17658] [[15051,1],7] creating OOB-TCP module for interface eth0
[cn4:17658] [[15051,1],7] oob:tcp:init adding 192.168.1.4 to our list of V4 connections
[cn4:17658] [[15051,1],7] TCP STARTUP
[cn4:17658] [[15051,1],7] attempting to bind to IPv4 port 0
[cn4:17658] [[15051,1],7] assigned IPv4 port 39807
[cn4:17658] mca:oob:select: Adding component to end
[cn4:17658] mca:oob:select: Found 1 active transports
[cn3:21245] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn1:13986] [[15051,0],0] message received 159 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 247 bytes from [[15051,0],1] for tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 263 bytes from [[15051,0],3] for tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],2] for tag 2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn5:13994] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn5:13994] [[15051,1],8] oob:tcp:init rejecting loopback interface lo
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 2016 BYTES ON SOCKET 9
[cn2:00789] [[15051,1],3] oob:tcp:init rejecting loopback interface lo
[cn2:00789] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn2:00789] [[15051,1],3] oob:tcp:init creating module for V4 address on interface eth0
[cn4:17657] mca: base: components_register: registering rml components
[cn4:17657] mca: base: components_register: found loaded component oob
[cn4:17657] mca: base: components_register: component oob has no register or open function
[cn3:21245] [[15051,1],4] oob:tcp:init rejecting loopback interface lo
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 482
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 482 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 442
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 442 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 304
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 304 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 88
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 88 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 482 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 442 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 304 bytes from [[15051,0],3] for tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] message received 88 bytes from [[15051,0],2] for tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 2459 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 263
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 102 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 263 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 187
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 187 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 190 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13994] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn5:13994] [[15051,1],8] oob:tcp:init creating module for V4 address on interface eth0
[cn5:13994] [[15051,1],8] creating OOB-TCP module for interface eth0
[cn5:13994] [[15051,1],8] oob:tcp:init adding 192.168.1.5 to our list of V4 connections
[cn5:13994] [[15051,1],8] TCP STARTUP
[cn5:13994] [[15051,1],8] attempting to bind to IPv4 port 0
[cn5:13994] [[15051,1],8] assigned IPv4 port 53909
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 186 BYTES ON SOCKET 9
[cn2:00789] [[15051,1],3] creating OOB-TCP module for interface eth0
[cn2:00789] [[15051,1],3] oob:tcp:init adding 192.168.1.2 to our list of V4 connections
[cn2:00789] [[15051,1],3] TCP STARTUP
[cn2:00789] [[15051,1],3] attempting to bind to IPv4 port 0
[cn2:00789] [[15051,1],3] assigned IPv4 port 34141
[cn2:00789] mca:oob:select: Adding component to end
[cn2:00789] mca:oob:select: Found 1 active transports
[cn4:17657] mca: base: components_open: opening rml components
[cn4:17657] mca: base: components_open: found loaded component oob
[cn4:17657] mca: base: components_open: component oob open function successful
[cn4:17657] orte_rml_base_select: initializing rml component oob
[cn3:21245] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 136
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 136 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 263
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 263 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 263 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 187 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] message received 136 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 263 bytes from [[15051,0],1] for tag 2
[cn4:17658] mca: base: components_register: registering rml components
[cn4:17658] mca: base: components_register: found loaded component oob
[cn4:17658] mca: base: components_register: component oob has no register or open function
[cn3:21245] [[15051,1],4] oob:tcp:init creating module for V4 address on interface eth0
[cn3:21245] [[15051,1],4] creating OOB-TCP module for interface eth0
[cn5:13994] mca:oob:select: Adding component to end
[cn5:13994] mca:oob:select: Found 1 active transports
[cn2:00789] mca: base: components_register: registering rml components
[cn2:00789] mca: base: components_register: found loaded component oob
[cn2:00789] mca: base: components_register: component oob has no register or open function
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 304
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 304 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 119 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 304
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 855 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 304 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 373
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 373 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 304 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 304 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 373 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 3276 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 855 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 276
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 276 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 172
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 172 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13995] mca: base: components_register: found loaded component tcp
[cn2:00789] mca: base: components_open: opening rml components
[cn2:00789] mca: base: components_open: found loaded component oob
[cn2:00789] mca: base: components_open: component oob open function successful
[cn2:00789] orte_rml_base_select: initializing rml component oob
[cn4:17658] mca: base: components_open: opening rml components
[cn4:17658] mca: base: components_open: found loaded component oob
[cn4:17658] mca: base: components_open: component oob open function successful
[cn4:17658] orte_rml_base_select: initializing rml component oob
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21245] [[15051,1],4] oob:tcp:init adding 192.168.1.3 to our list of V4 connections
[cn3:21245] [[15051,1],4] TCP STARTUP
[cn3:21245] [[15051,1],4] attempting to bind to IPv4 port 0
[cn3:21245] [[15051,1],4] assigned IPv4 port 49283
[cn3:21245] mca:oob:select: Adding component to end
[cn3:21245] mca:oob:select: Found 1 active transports
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 104
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 104 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 276 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 172 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 104 bytes from [[15051,0],1] for tag 2
[cn4:17657] mca: base: components_register: registering routed components
[cn4:17657] mca: base: components_register: found loaded component binomial
[cn4:17657] mca: base: components_register: component binomial has no register or open function
[cn3:21246] mca: base: components_register: registering oob components
[cn3:21246] mca: base: components_register: found loaded component tcp
[cn5:13995] mca: base: components_register: component tcp register function successful
[cn2:00788] mca: base: components_register: registering routed components
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 401
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 401 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 202
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 202 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 532
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 532 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 401 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 202 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 532 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],2] for tag 2
[cn5:13995] mca: base: components_open: opening oob components
[cn5:13995] mca: base: components_open: found loaded component tcp
[cn5:13995] mca: base: components_open: component tcp open function successful
[cn5:13995] mca:oob:select: checking available component tcp
[cn5:13995] mca:oob:select: Querying component [tcp]
[cn5:13995] oob:tcp: component_available called
[cn2:00788] mca: base: components_register: found loaded component binomial
[cn2:00788] mca: base: components_register: component binomial has no register or open function
[cn4:17657] mca: base: components_register: found loaded component debruijn
[cn4:17657] mca: base: components_register: component debruijn has no register or open function
[cn4:17657] mca: base: components_register: found loaded component direct
[cn4:17657] mca: base: components_register: component direct has no register or open function
[cn4:17657] mca: base: components_register: found loaded component radix
[cn4:17657] mca: base: components_register: component radix register function successful
[cn3:21246] mca: base: components_register: component tcp register function successful
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 88
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 88 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 443
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 443 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1280
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 1280 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 401
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 401 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 88 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 443 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 1280 bytes from [[15051,0],3] for tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] message received 401 bytes from [[15051,0],2] for tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13995] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00788] mca: base: components_register: found loaded component debruijn
[cn2:00788] mca: base: components_register: component debruijn has no register or open function
[cn2:00788] mca: base: components_register: found loaded component direct
[cn2:00788] mca: base: components_register: component direct has no register or open function
[cn2:00788] mca: base: components_register: found loaded component radix
[cn4:17657] mca: base: components_open: opening routed components
[cn4:17657] mca: base: components_open: found loaded component binomial
[cn4:17657] mca: base: components_open: found loaded component debruijn
[cn4:17657] mca: base: components_open: found loaded component direct
[cn4:17657] mca: base: components_open: found loaded component radix
[cn4:17657] mca:base:select: Auto-selecting routed components
[cn4:17657] mca:base:select:(routed) Querying component [binomial]
[cn4:17657] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn4:17657] mca:base:select:(routed) Querying component [debruijn]
[cn4:17657] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn4:17657] mca:base:select:(routed) Querying component [direct]
[cn4:17657] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn4:17657] mca:base:select:(routed) Querying component [radix]
[cn4:17657] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn4:17657] mca:base:select:(routed) Selected component [binomial]
[cn4:17657] mca: base: close: unloading component debruijn
[cn4:17657] mca: base: close: unloading component direct
[cn4:17657] mca: base: close: unloading component radix
[cn3:21246] mca: base: components_open: opening oob components
[cn3:21246] mca: base: components_open: found loaded component tcp
[cn3:21246] mca: base: components_open: component tcp open function successful
[cn3:21246] mca:oob:select: checking available component tcp
[cn3:21246] mca:oob:select: Querying component [tcp]
[cn3:21246] oob:tcp: component_available called
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 319 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 188 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 247
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 247 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 319 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1256
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 1256 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 2443 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 108
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 108 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 88
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 88 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 247 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 1256 bytes from [[15051,0],1] for tag 2
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 108 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 245 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] message received 88 bytes from [[15051,0],2] for tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn5:13995] [[15051,1],9] oob:tcp:init rejecting loopback interface lo
[cn5:13995] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn5:13995] [[15051,1],9] oob:tcp:init creating module for V4 address on interface eth0
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00788] mca: base: components_register: component radix register function successful
[cn2:00788] mca: base: components_open: opening routed components
[cn2:00788] mca: base: components_open: found loaded component binomial
[cn2:00788] mca: base: components_open: found loaded component debruijn
[cn2:00788] mca: base: components_open: found loaded component direct
[cn2:00788] mca: base: components_open: found loaded component radix
[cn2:00788] mca:base:select: Auto-selecting routed components
[cn2:00788] mca:base:select:(routed) Querying component [binomial]
[cn2:00788] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn2:00788] mca:base:select:(routed) Querying component [debruijn]
[cn2:00788] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn2:00788] mca:base:select:(routed) Querying component [direct]
[cn2:00788] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn2:00788] mca:base:select:(routed) Querying component [radix]
[cn2:00788] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn2:00788] mca:base:select:(routed) Selected component [binomial]
[cn2:00788] mca: base: close: unloading component debruijn
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17657] orte_routed_base_select: initializing selected component binomial
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn3:21246] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 442
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 442 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 184 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 221
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 221 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 448
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 103 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 448 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 537
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 537 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 442 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 221 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 448 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 537 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 104
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 104 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 360
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 360 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] [[15051,1],9] creating OOB-TCP module for interface eth0
[cn5:13995] [[15051,1],9] oob:tcp:init adding 192.168.1.5 to our list of V4 connections
[cn5:13995] [[15051,1],9] TCP STARTUP
[cn5:13995] [[15051,1],9] attempting to bind to IPv4 port 0
[cn5:13995] [[15051,1],9] assigned IPv4 port 54485
[cn5:13995] mca:oob:select: Adding component to end
[cn5:13995] mca:oob:select: Found 1 active transports
[cn2:00788] mca: base: close: unloading component direct
[cn2:00788] mca: base: close: unloading component radix
[cn2:00788] orte_routed_base_select: initializing selected component binomial
[cn4:17658] mca: base: components_register: registering routed components
[cn4:17658] mca: base: components_register: found loaded component binomial
[cn4:17658] mca: base: components_register: component binomial has no register or open function
[cn4:17658] mca: base: components_register: found loaded component debruijn
[cn4:17658] mca: base: components_register: component debruijn has no register or open function
[cn3:21245] mca: base: components_register: registering rml components
[cn3:21245] mca: base: components_register: found loaded component oob
[cn3:21245] mca: base: components_register: component oob has no register or open function
[cn3:21245] mca: base: components_open: opening rml components
[cn3:21245] mca: base: components_open: found loaded component oob
[cn3:21245] mca: base: components_open: component oob open function successful
[cn3:21245] orte_rml_base_select: initializing rml component oob
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 263
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 263 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 104 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 360 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] message received 263 bytes from [[15051,0],4] for tag 2
[cn2:00789] mca: base: components_register: registering routed components
[cn4:17658] mca: base: components_register: found loaded component direct
[cn4:17658] mca: base: components_register: component direct has no register or open function
[cn4:17658] mca: base: components_register: found loaded component radix
[cn4:17658] mca: base: components_register: component radix register function successful
[cn3:21246] [[15051,1],5] oob:tcp:init rejecting loopback interface lo
[cn5:13994] mca: base: components_register: registering rml components
[cn5:13994] mca: base: components_register: found loaded component oob
[cn5:13994] mca: base: components_register: component oob has no register or open function
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 304
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 304 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 264
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 264 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1358
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 1358 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 245
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 245 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 304 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 264 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 1358 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 245 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 263
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 263 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 286
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 286 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 176
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 176 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 156
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 156 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] mca: base: components_open: opening rml components
[cn5:13994] mca: base: components_open: found loaded component oob
[cn5:13994] mca: base: components_open: component oob open function successful
[cn5:13994] orte_rml_base_select: initializing rml component oob
[cn2:00788] mca: base: components_register: registering dstore components
[cn2:00788] mca: base: components_register: found loaded component hash
[cn2:00788] mca: base: components_register: component hash register function successful
[cn4:17658] mca: base: components_open: opening routed components
[cn4:17658] mca: base: components_open: found loaded component binomial
[cn4:17658] mca: base: components_open: found loaded component debruijn
[cn4:17658] mca: base: components_open: found loaded component direct
[cn4:17658] mca: base: components_open: found loaded component radix
[cn4:17658] mca:base:select: Auto-selecting routed components
[cn4:17658] mca:base:select:(routed) Querying component [binomial]
[cn4:17658] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn4:17658] mca:base:select:(routed) Querying component [debruijn]
[cn4:17658] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn4:17658] mca:base:select:(routed) Querying component [direct]
[cn4:17658] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn4:17658] mca:base:select:(routed) Querying component [radix]
[cn4:17658] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn4:17658] mca:base:select:(routed) Selected component [binomial]
[cn4:17658] mca: base: close: unloading component debruijn
[cn4:17658] mca: base: close: unloading component direct
[cn4:17658] mca: base: close: unloading component radix
[cn4:17658] orte_routed_base_select: initializing selected component binomial
[cn3:21246] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn3:21246] [[15051,1],5] oob:tcp:init creating module for V4 address on interface eth0
[cn3:21246] [[15051,1],5] creating OOB-TCP module for interface eth0
[cn1:13986] [[15051,0],0] message received 263 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 286 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 176 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 156 bytes from [[15051,0],2] for tag 2
[cn5:13995] mca: base: components_register: registering rml components
[cn5:13995] mca: base: components_register: found loaded component oob
[cn5:13995] mca: base: components_register: component oob has no register or open function
[cn2:00788] mca: base: components_open: opening dstore components
[cn2:00788] mca: base: components_open: found loaded component hash
[cn2:00788] mca:dstore:select: checking available component hash
[cn2:00788] mca:dstore:select: Querying component [hash]
[cn4:17657] mca: base: components_register: registering dstore components
[cn4:17657] mca: base: components_register: found loaded component hash
[cn3:21246] [[15051,1],5] oob:tcp:init adding 192.168.1.3 to our list of V4 connections
[cn3:21246] [[15051,1],5] TCP STARTUP
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 304
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 304 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 106
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 106 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 374
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 374 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 247
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 247 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 304 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 106 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 374 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 247 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_open: opening rml components
[cn5:13995] mca: base: components_open: found loaded component oob
[cn5:13995] mca: base: components_open: component oob open function successful
[cn5:13995] orte_rml_base_select: initializing rml component oob
[cn2:00789] mca: base: components_register: found loaded component binomial
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 811 BYTES ON SOCKET 9
[cn4:17657] mca: base: components_register: component hash register function successful
[cn4:17657] mca: base: components_open: opening dstore components
[cn4:17657] mca: base: components_open: found loaded component hash
[cn4:17657] mca:dstore:select: checking available component hash
[cn4:17657] mca:dstore:select: Querying component [hash]
[cn3:21246] [[15051,1],5] attempting to bind to IPv4 port 0
[cn3:21246] [[15051,1],5] assigned IPv4 port 58679
[cn3:21246] mca:oob:select: Adding component to end
[cn3:21246] mca:oob:select: Found 1 active transports
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 276
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 276 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 126
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 506 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 104
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 104 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],2] for tag 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] message received 276 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 126 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 104 bytes from [[15051,0],3] for tag 2
[cn3:21246] mca: base: components_register: registering rml components
[cn5:13994] mca: base: components_register: registering routed components
[cn5:13994] mca: base: components_register: found loaded component binomial
[cn5:13994] mca: base: components_register: component binomial has no register or open function
[cn2:00789] mca: base: components_register: component binomial has no register or open function
[cn4:17658] mca: base: components_register: registering dstore components
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 370
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 370 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 443
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 116 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 443 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 190
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 190 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 464 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 192
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 192 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 464 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] message received 370 bytes from [[15051,0],4] for tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 443 bytes from [[15051,0],1] for tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] message received 190 bytes from [[15051,0],3] for tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 192 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 273 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 402 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1256
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 1256 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 286
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 2016 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 286 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 304
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 2016 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 304 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13994] mca: base: components_register: found loaded component debruijn
[cn5:13994] mca: base: components_register: component debruijn has no register or open function
[cn5:13994] mca: base: components_register: found loaded component direct
[cn5:13994] mca: base: components_register: component direct has no register or open function
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00789] mca: base: components_register: found loaded component debruijn
[cn2:00789] mca: base: components_register: component debruijn has no register or open function
[cn2:00789] mca: base: components_register: found loaded component direct
[cn2:00789] mca: base: components_register: component direct has no register or open function
[cn2:00789] mca: base: components_register: found loaded component radix
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17658] mca: base: components_register: found loaded component hash
[cn4:17658] mca: base: components_register: component hash register function successful
[cn3:21246] mca: base: components_register: found loaded component oob
[cn3:21246] mca: base: components_register: component oob has no register or open function
[cn1:14002] mca: base: components_register: registering ess components
[cn1:14002] mca: base: components_register: found loaded component env
[cn1:14002] mca: base: components_register: component env has no register or open function
[cn1:14002] mca: base: components_open: opening ess components
[cn1:14002] mca: base: components_open: found loaded component env
[cn1:14002] mca: base: components_open: component env open function successful
[cn1:14002] mca:base:select: Auto-selecting ess components
[cn1:14002] mca:base:select:(  ess) Querying component [env]
[cn1:14002] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn1:14002] mca:base:select:(  ess) Selected component [env]
[cn1:14002] ess:env set name to [[15051,1],1]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 756
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 321 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 756 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 1256 bytes from [[15051,0],1] for tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 286 bytes from [[15051,0],3] for tag 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 409 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 304 bytes from [[15051,0],2] for tag 2
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] message received 756 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 221
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 221 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 267
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 267 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 104
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 104 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00789] mca: base: components_register: component radix register function successful
[cn2:00789] mca: base: components_open: opening routed components
[cn2:00789] mca: base: components_open: found loaded component binomial
[cn2:00789] mca: base: components_open: found loaded component debruijn
[cn2:00789] mca: base: components_open: found loaded component direct
[cn2:00789] mca: base: components_open: found loaded component radix
[cn2:00789] mca:base:select: Auto-selecting routed components
[cn2:00789] mca:base:select:(routed) Querying component [binomial]
[cn2:00789] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn2:00789] mca:base:select:(routed) Querying component [debruijn]
[cn2:00789] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn2:00789] mca:base:select:(routed) Querying component [direct]
[cn2:00789] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn2:00789] mca:base:select:(routed) Querying component [radix]
[cn2:00789] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn2:00789] mca:base:select:(routed) Selected component [binomial]
[cn2:00789] mca: base: close: unloading component debruijn
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 255 BYTES ON SOCKET 9
[cn4:17658] mca: base: components_open: opening dstore components
[cn4:17658] mca: base: components_open: found loaded component hash
[cn4:17658] mca:dstore:select: checking available component hash
[cn4:17658] mca:dstore:select: Querying component [hash]
[cn3:21246] mca: base: components_open: opening rml components
[cn3:21246] mca: base: components_open: found loaded component oob
[cn3:21246] mca: base: components_open: component oob open function successful
[cn3:21246] orte_rml_base_select: initializing rml component oob
[cn5:13994] mca: base: components_register: found loaded component radix
[cn5:13994] mca: base: components_register: component radix register function successful
[cn5:13994] mca: base: components_open: opening routed components
[cn5:13994] mca: base: components_open: found loaded component binomial
[cn5:13994] mca: base: components_open: found loaded component debruijn
[cn5:13994] mca: base: components_open: found loaded component direct
[cn5:13994] mca: base: components_open: found loaded component radix
[cn5:13994] mca:base:select: Auto-selecting routed components
[cn5:13994] mca:base:select:(routed) Querying component [binomial]
[cn5:13994] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 794
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 245 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 794 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] message received 221 bytes from [[15051,0],1] for tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 267 bytes from [[15051,0],3] for tag 2
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 104 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 794 bytes from [[15051,0],4] for tag 2
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 276
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 276 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 267
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 267 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 107 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 258 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 738
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 738 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 202
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 202 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00789] mca: base: close: unloading component direct
[cn2:00789] mca: base: close: unloading component radix
[cn2:00789] orte_routed_base_select: initializing selected component binomial
[cn4:17657] mca: base: components_register: registering grpcomm components
[cn4:17657] mca: base: components_register: found loaded component bad
[cn4:17657] mca: base: components_register: component bad has no register or open function
[cn3:21245] mca: base: components_register: registering routed components
[cn5:13994] mca:base:select:(routed) Querying component [debruijn]
[cn5:13994] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn5:13994] mca:base:select:(routed) Querying component [direct]
[cn5:13994] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn5:13994] mca:base:select:(routed) Querying component [radix]
[cn5:13994] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn5:13994] mca:base:select:(routed) Selected component [binomial]
[cn5:13994] mca: base: close: unloading component debruijn
[cn5:13994] mca: base: close: unloading component direct
[cn5:13994] mca: base: close: unloading component radix
[cn5:13994] orte_routed_base_select: initializing selected component binomial
[cn1:13986] [[15051,0],0] message received 276 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 267 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 738 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 202 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13995] mca: base: components_register: registering routed components
[cn5:13995] mca: base: components_register: found loaded component binomial
[cn5:13995] mca: base: components_register: component binomial has no register or open function
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 319 BYTES ON SOCKET 9
[cn2:00788] mca: base: components_register: registering grpcomm components
[cn2:00788] mca: base: components_register: found loaded component bad
[cn2:00788] mca: base: components_register: component bad has no register or open function
[cn4:17657] mca: base: components_open: opening grpcomm components
[cn4:17657] mca: base: components_open: found loaded component bad
[cn4:17657] mca: base: components_open: component bad open function successful
[cn4:17657] mca:base:select: Auto-selecting grpcomm components
[cn4:17657] mca:base:select:(grpcomm) Querying component [bad]
[cn4:17657] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn4:17657] mca:base:select:(grpcomm) Selected component [bad]
[cn4:17657] [[15051,1],6] grpcomm:base:receive start comm
[cn4:17657] [[15051,1],6] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn4:17657] [[15051,1],6] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn3:21245] mca: base: components_register: found loaded component binomial
[cn3:21245] mca: base: components_register: component binomial has no register or open function
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 190 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 443
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 443 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 738
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 738 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 242 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 190
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 190 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 443
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 443 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 443 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 738 bytes from [[15051,0],1] for tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 119 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 190 bytes from [[15051,0],3] for tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 443 bytes from [[15051,0],2] for tag 2
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 187 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1313
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 103 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1313 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 190
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 190 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 105
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 105 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1256
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1256 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_register: found loaded component debruijn
[cn5:13995] mca: base: components_register: component debruijn has no register or open function
[cn5:13995] mca: base: components_register: found loaded component direct
[cn5:13995] mca: base: components_register: component direct has no register or open function
[cn5:13995] mca: base: components_register: found loaded component radix
[cn2:00788] mca: base: components_open: opening grpcomm components
[cn2:00788] mca: base: components_open: found loaded component bad
[cn2:00788] mca: base: components_open: component bad open function successful
[cn2:00788] mca:base:select: Auto-selecting grpcomm components
[cn2:00788] mca:base:select:(grpcomm) Querying component [bad]
[cn2:00788] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn2:00788] mca:base:select:(grpcomm) Selected component [bad]
[cn2:00788] [[15051,1],2] grpcomm:base:receive start comm
[cn2:00788] [[15051,1],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn2:00788] [[15051,1],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn4:17657] [[15051,1],6] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn4:17657] [[15051,1],6] setting up session dir with
	tmpdir: /tmp
	host cn4
[cn3:21245] mca: base: components_register: found loaded component debruijn
[cn3:21245] mca: base: components_register: component debruijn has no register or open function
[cn3:21245] mca: base: components_register: found loaded component direct
[cn3:21245] mca: base: components_register: component direct has no register or open function
[cn3:21245] mca: base: components_register: found loaded component radix
[cn1:13986] [[15051,0],0] message received 1313 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 190 bytes from [[15051,0],1] for tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] message received 105 bytes from [[15051,0],3] for tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 1256 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 182 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 104
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 104 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 158 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 221
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 99 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 221 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_register: component radix register function successful
[cn5:13995] mca: base: components_open: opening routed components
[cn5:13995] mca: base: components_open: found loaded component binomial
[cn5:13995] mca: base: components_open: found loaded component debruijn
[cn5:13995] mca: base: components_open: found loaded component direct
[cn5:13995] mca: base: components_open: found loaded component radix
[cn5:13995] mca:base:select: Auto-selecting routed components
[cn5:13995] mca:base:select:(routed) Querying component [binomial]
[cn5:13995] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn5:13995] mca:base:select:(routed) Querying component [debruijn]
[cn5:13995] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn5:13995] mca:base:select:(routed) Querying component [direct]
[cn5:13995] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn5:13995] mca:base:select:(routed) Querying component [radix]
[cn5:13995] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn5:13995] mca:base:select:(routed) Selected component [binomial]
[cn5:13995] mca: base: close: unloading component debruijn
[cn5:13995] mca: base: close: unloading component direct
[cn2:00788] [[15051,1],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn2:00788] [[15051,1],2] setting up session dir with
	tmpdir: /tmp
	host cn2
[cn4:17658] mca: base: components_register: registering grpcomm components
[cn3:21245] mca: base: components_register: component radix register function successful
[cn3:21245] mca: base: components_open: opening routed components
[cn3:21245] mca: base: components_open: found loaded component binomial
[cn3:21245] mca: base: components_open: found loaded component debruijn
[cn3:21245] mca: base: components_open: found loaded component direct
[cn3:21245] mca: base: components_open: found loaded component radix
[cn3:21245] mca:base:select: Auto-selecting routed components
[cn3:21245] mca:base:select:(routed) Querying component [binomial]
[cn3:21245] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn3:21245] mca:base:select:(routed) Querying component [debruijn]
[cn3:21245] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn3:21245] mca:base:select:(routed) Querying component [direct]
[cn3:21245] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn3:21245] mca:base:select:(routed) Querying component [radix]
[cn3:21245] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn3:21245] mca:base:select:(routed) Selected component [binomial]
[cn3:21245] mca: base: close: unloading component debruijn
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 164
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 164 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 104 bytes from [[15051,0],1] for tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 541 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],3] for tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] message received 221 bytes from [[15051,0],2] for tag 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] message received 164 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 104
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 104 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 186 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 409 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 102
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 102 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 419
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 419 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 176
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 176 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00789] mca: base: components_register: registering dstore components
[cn4:17658] mca: base: components_register: found loaded component bad
[cn3:21245] mca: base: close: unloading component direct
[cn3:21245] mca: base: close: unloading component radix
[cn3:21245] orte_routed_base_select: initializing selected component binomial
[cn5:13995] mca: base: close: unloading component radix
[cn5:13995] orte_routed_base_select: initializing selected component binomial
[cn1:13986] [[15051,0],0] message received 104 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 102 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 419 bytes from [[15051,0],3] for tag 2
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] message received 176 bytes from [[15051,0],2] for tag 2
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 102
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 102 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 187 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 290
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 290 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 99 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 121
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 121 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 374
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 374 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] mca: base: components_register: registering dstore components
[cn2:00789] mca: base: components_register: found loaded component hash
[cn4:17657] [[15051,1],6] posting recv
[cn4:17657] [[15051,1],6] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn4:17657] [[15051,1],6] posting recv
[cn4:17657] [[15051,1],6] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn4:17657] [[15051,1],6] posting recv
[cn4:17657] [[15051,1],6] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn3:21245] mca: base: components_register: registering dstore components
[cn3:21245] mca: base: components_register: found loaded component hash
[cn1:13986] [[15051,0],0] message received 102 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 290 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 121 bytes from [[15051,0],3] for tag 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 374 bytes from [[15051,0],2] for tag 2
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13994] mca: base: components_register: found loaded component hash
[cn2:00788] [[15051,1],2] posting recv
[cn2:00788] [[15051,1],2] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn2:00788] [[15051,1],2] posting recv
[cn2:00788] [[15051,1],2] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17658] mca: base: components_register: component bad has no register or open function
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21245] mca: base: components_register: component hash register function successful
[cn3:21245] mca: base: components_open: opening dstore components
[cn3:21245] mca: base: components_open: found loaded component hash
[cn3:21245] mca:dstore:select: checking available component hash
[cn3:21245] mca:dstore:select: Querying component [hash]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 187 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 264
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 264 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 326 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 159
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 159 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 336
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 336 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 104
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 104 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 264 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 159 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 336 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 104 bytes from [[15051,0],2] for tag 2
[cn5:13995] mca: base: components_register: registering dstore components
[cn5:13995] mca: base: components_register: found loaded component hash
[cn5:13995] mca: base: components_register: component hash register function successful
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00788] [[15051,1],2] posting recv
[cn2:00788] [[15051,1],2] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17657] procdir: /tmp/openmpi-sessions-artpol@cn4_0/15051/1/6
[cn4:17657] jobdir: /tmp/openmpi-sessions-artpol@cn4_0/15051/1
[cn4:17657] top: openmpi-sessions-artpol@cn4_0
[cn4:17657] tmp: /tmp
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn3:21246] mca: base: components_register: registering routed components
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 286
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 286 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 118
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 118 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 158 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 186 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 945
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 945 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 106
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 107 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 106 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 286 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] message received 118 bytes from [[15051,0],1] for tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 319 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 945 bytes from [[15051,0],3] for tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 106 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 245 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13995] mca: base: components_open: opening dstore components
[cn5:13995] mca: base: components_open: found loaded component hash
[cn5:13995] mca:dstore:select: checking available component hash
[cn5:13995] mca:dstore:select: Querying component [hash]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00789] mca: base: components_register: component hash register function successful
[cn4:17657] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 26756186113
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 26756186113
[cn4:17657] [[15051,1],6] routed_binomial: init routes for proc job [15051,1]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
	daemon uri 986382336.3;tcp://192.168.1.4:54546
[cn4:17657] [[15051,1],6] [[15051,0],3] with daemon registering
[cn4:17657] [[15051,1],6] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn4:17657] [[15051,1],6] rml_send_buffer to peer [[15051,0],3] at tag 1
[cn4:17657] [[15051,1],6] registering sync waiting for ack
[cn4:17657] [[15051,1],6] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn3:21246] mca: base: components_register: found loaded component binomial
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 118
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 118 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 335
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 335 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 551
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 551 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 242 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 267
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 267 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 115 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] message received 118 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] message received 335 bytes from [[15051,0],1] for tag 2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] message received 551 bytes from [[15051,0],3] for tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] message received 267 bytes from [[15051,0],2] for tag 2
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 96
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 187 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 96 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 661
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 661 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1155
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 1155 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 187 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 738
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 738 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13994] mca: base: components_register: component hash register function successful
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00788] procdir: /tmp/openmpi-sessions-artpol@cn2_0/15051/1/2
[cn2:00788] jobdir: /tmp/openmpi-sessions-artpol@cn2_0/15051/1
[cn2:00788] top: openmpi-sessions-artpol@cn2_0
[cn2:00788] tmp: /tmp
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn4:17657] [[15051,1],6]: set_addr to uri 986382336.3;tcp://192.168.1.4:54546
[cn4:17657] [[15051,1],6]:set_addr checking if peer [[15051,0],3] is reachable via component tcp
[cn4:17657] [[15051,1],6] oob:tcp: working peer [[15051,0],3] address tcp://192.168.1.4:54546
[cn4:17657] [[15051,1],6] PEER [[15051,0],3] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17657] [[15051,1],6] PASSING ADDR 192.168.1.4 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn4:17657] [[15051,1],6]:tcp set addr for peer [[15051,0],3]
[cn3:21245] mca: base: components_register: registering grpcomm components
[cn3:21245] mca: base: components_register: found loaded component bad
[cn3:21245] mca: base: components_register: component bad has no register or open function
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 180 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 96 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 661 bytes from [[15051,0],1] for tag 2
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0] message received 1155 bytes from [[15051,0],3] for tag 2
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 738 bytes from [[15051,0],2] for tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 95 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13994] mca: base: components_open: opening dstore components
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn2:00788] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 9576316929
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 9576316929
[cn2:00788] [[15051,1],2] routed_binomial: init routes for proc job [15051,1]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
	daemon uri 986382336.1;tcp://192.168.1.2:41722
[cn2:00788] [[15051,1],2] [[15051,0],1] with daemon registering
[cn2:00788] [[15051,1],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn4:17657] [[15051,1],6]: peer [[15051,0],3] is reachable via component tcp
[cn4:17657] [[15051,1],6] posting recv
[cn4:17657] [[15051,1],6] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn4:17657] [[15051,1],6] rml_send_msg to peer [[15051,0],3] at tag 1
[cn4:17657] [[15051,1],6] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17657] [[15051,1],6]:tcp:processing set_peer cmd for interface eth0
[cn4:17657] [[15051,1],6] SET_PEER ADDING PEER [[15051,0],3]
[cn4:17657] [[15051,1],6] set_peer: peer [[15051,0],3] is listening on interface eth0 net 192.168.1.4 port 54546
[cn4:17657] [[15051,1],6] oob:base:send to target [[15051,0],3]
[cn4:17657] [[15051,1],6] oob:tcp:send_nb to peer [[15051,0],3]:1
[cn4:17657] [[15051,1],6] tcp:send_nb to peer [[15051,0],3]
[cn4:17657] [[15051,1],6]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],3]
[cn4:17657] [[15051,1],6]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],3]:1
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21245] mca: base: components_open: opening grpcomm components
[cn3:21245] mca: base: components_open: found loaded component bad
[cn3:21245] mca: base: components_open: component bad open function successful
[cn3:21245] mca:base:select: Auto-selecting grpcomm components
[cn3:21245] mca:base:select:(grpcomm) Querying component [bad]
[cn3:21245] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn3:21245] mca:base:select:(grpcomm) Selected component [bad]
[cn3:21245] [[15051,1],4] grpcomm:base:receive start comm
[cn3:21245] [[15051,1],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn3:21245] [[15051,1],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 107 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 220
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 220 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 683
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 683 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 908
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 908 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 190
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 190 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 220 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 683 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 908 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 190 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_open: found loaded component hash
[cn5:13994] mca:dstore:select: checking available component hash
[cn5:13994] mca:dstore:select: Querying component [hash]
[cn2:00788] [[15051,1],2] rml_send_buffer to peer [[15051,0],1] at tag 1
[cn2:00788] [[15051,1],2] registering sync waiting for ack
[cn2:00788] [[15051,1],2] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn2:00788] [[15051,1],2]: set_addr to uri 986382336.1;tcp://192.168.1.2:41722
[cn2:00788] [[15051,1],2]:set_addr checking if peer [[15051,0],1] is reachable via component tcp
[cn2:00788] [[15051,1],2] oob:tcp: working peer [[15051,0],1] address tcp://192.168.1.2:41722
[cn2:00788] [[15051,1],2] PEER [[15051,0],1] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17657] [[15051,1],6] routed_binomial_get([[15051,0],3]) --> [[15051,0],3]
[cn4:17657] [[15051,1],6]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],3]
[cn4:17657] [[15051,1],6] tcp:send_nb: initiating connection to [[15051,0],3]
[cn4:17657] [[15051,1],6]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],3]
[cn4:17657] [[15051,1],6] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],3] via interface eth0
[cn4:17657] [[15051,1],6] oob:tcp:peer creating socket to [[15051,0],3]
[cn4:17657] [[15051,1],6] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],3] via interface eth0 on socket 11
[cn4:17657] [[15051,1],6] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],3] on 192.168.1.4:54546 - 0 retries
[cn3:21245] [[15051,1],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn3:21245] [[15051,1],4] setting up session dir with
	tmpdir: /tmp
	host cn3
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 105
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 105 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 829
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 829 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 678
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 678 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 105 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 829 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 678 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 126 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 192
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 192 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 936
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 936 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 319 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 260
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 260 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 242 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 106
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 106 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_register: registering grpcomm components
[cn2:00788] [[15051,1],2] PASSING ADDR 192.168.1.2 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00788] [[15051,1],2]:tcp set addr for peer [[15051,0],1]
[cn2:00788] [[15051,1],2]: peer [[15051,0],1] is reachable via component tcp
[cn2:00788] [[15051,1],2] posting recv
[cn2:00788] [[15051,1],2] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn2:00788] [[15051,1],2] rml_send_msg to peer [[15051,0],1] at tag 1
[cn2:00788] [[15051,1],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00788] [[15051,1],2]:tcp:processing set_peer cmd for interface eth0
[cn2:00788] [[15051,1],2] SET_PEER ADDING PEER [[15051,0],1]
[cn2:00788] [[15051,1],2] set_peer: peer [[15051,0],1] is listening on interface eth0 net 192.168.1.2 port 41722
[cn4:17657] [[15051,1],6] waiting for connect completion to [[15051,0],3] - activating send event
[cn4:17657] [[15051,1],6] tcp:send_handler called to send to peer [[15051,0],3]
[cn4:17657] [[15051,1],6] tcp:send_handler CONNECTING
[cn4:17657] [[15051,1],6]:tcp:complete_connect called for peer [[15051,0],3] on socket 11
[cn4:17657] [[15051,1],6] tcp_peer_complete_connect: sending ack to [[15051,0],3]
[cn4:17657] [[15051,1],6] SEND CONNECT ACK
[cn4:17657] fetching data from INTERNAL dstore
[cn4:17657] dstore:hash:fetch: searching for key opal.cred on proc 26756186113
[cn4:17657] dstore_hash:fetch key opal.cred for proc 26756186113 not found
[cn3:21246] mca: base: components_register: component binomial has no register or open function
[cn1:13986] [[15051,0],0] message received 192 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 936 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 260 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 106 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 738
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 738 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 655
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 655 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1217
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 1217 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 419
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 419 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_register: found loaded component bad
[cn5:13995] mca: base: components_register: component bad has no register or open function
[cn2:00788] [[15051,1],2] oob:base:send to target [[15051,0],1]
[cn2:00788] [[15051,1],2] oob:tcp:send_nb to peer [[15051,0],1]:1
[cn2:00788] [[15051,1],2] tcp:send_nb to peer [[15051,0],1]
[cn2:00788] [[15051,1],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],1]
[cn2:00788] [[15051,1],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],1]:1
[cn2:00788] [[15051,1],2] routed_binomial_get([[15051,0],1]) --> [[15051,0],1]
[cn2:00788] [[15051,1],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],1]
[cn2:00788] [[15051,1],2] tcp:send_nb: initiating connection to [[15051,0],1]
[cn2:00788] [[15051,1],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],1]
[cn4:17657] [[15051,1],6] send blocking of 47 bytes to socket 11
[cn4:17657] [[15051,1],6] blocking send complete to socket 11
[cn4:17657] [[15051,1],6] tcp_peer_complete_connect: setting read event on connection to [[15051,0],3]
[cn3:21246] mca: base: components_register: found loaded component debruijn
[cn1:13986] [[15051,0],0] message received 738 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 655 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 1217 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 419 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 190
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 190 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 728
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 728 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 275
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 275 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_open: opening grpcomm components
[cn5:13995] mca: base: components_open: found loaded component bad
[cn5:13995] mca: base: components_open: component bad open function successful
[cn5:13995] mca:base:select: Auto-selecting grpcomm components
[cn5:13995] mca:base:select:(grpcomm) Querying component [bad]
[cn5:13995] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn5:13995] mca:base:select:(grpcomm) Selected component [bad]
[cn5:13995] [[15051,1],9] grpcomm:base:receive start comm
[cn5:13995] [[15051,1],9] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn5:13995] [[15051,1],9] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn2:00788] [[15051,1],2] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],1] via interface eth0
[cn2:00788] [[15051,1],2] oob:tcp:peer creating socket to [[15051,0],1]
[cn2:00788] [[15051,1],2] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],1] via interface eth0 on socket 11
[cn2:00788] [[15051,1],2] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],1] on 192.168.1.2:41722 - 0 retries
[cn2:00788] [[15051,1],2] waiting for connect completion to [[15051,0],1] - activating send event
[cn2:00788] [[15051,1],2] tcp:send_handler called to send to peer [[15051,0],1]
[cn4:17657] [[15051,1],6]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17657] [[15051,1],6] RECV CONNECT ACK FROM [[15051,0],3] ON SOCKET 11
[cn4:17657] [[15051,1],6] waiting for connect ack from [[15051,0],3]
[cn4:17657] [[15051,1],6] connect ack received from [[15051,0],3]
[cn4:17657] [[15051,1],6] connect-ack recvd from [[15051,0],3]
[cn4:17657] [[15051,1],6] connect-ack header from [[15051,0],3] is okay
[cn4:17657] [[15051,1],6] waiting for connect ack from [[15051,0],3]
[cn4:17657] [[15051,1],6] connect ack received from [[15051,0],3]
[cn4:17657] [[15051,1],6] connect-ack version from [[15051,0],3] matches ours
[cn4:17657] [[15051,1],6] connect-ack [[15051,0],3] authenticated
[cn4:17657] [[15051,1],6]-[[15051,0],3] tcp_peer_connected on socket 11
[cn4:17657] [[15051,1],6]-[[15051,0],3] connected: 192.168.1.4 - 192.168.1.4 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn4:17657] [[15051,1],6]:tcp:recv:handler starting send/recv events
[cn4:17657] [[15051,1],6] tcp:set_module called for peer [[15051,0],3]
[cn4:17657] [[15051,1],6] tcp:send_handler called to send to peer [[15051,0],3]
[cn4:17657] [[15051,1],6] tcp:send_handler SENDING TO [[15051,0],3]
[cn3:21245] [[15051,1],4] posting recv
[cn3:21245] [[15051,1],4] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn3:21245] [[15051,1],4] posting recv
[cn3:21245] [[15051,1],4] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn3:21245] [[15051,1],4] posting recv
[cn3:21245] [[15051,1],4] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] message received 190 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 728 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 275 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 126 bytes from [[15051,0],2] for tag 2
[cn5:13995] [[15051,1],9] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn5:13995] [[15051,1],9] setting up session dir with
	tmpdir: /tmp
	host cn5
[cn2:00788] [[15051,1],2] tcp:send_handler CONNECTING
[cn2:00788] [[15051,1],2]:tcp:complete_connect called for peer [[15051,0],1] on socket 11
[cn2:00788] [[15051,1],2] tcp_peer_complete_connect: sending ack to [[15051,0],1]
[cn2:00788] [[15051,1],2] SEND CONNECT ACK
[cn2:00788] fetching data from INTERNAL dstore
[cn2:00788] dstore:hash:fetch: searching for key opal.cred on proc 9576316929
[cn2:00788] dstore_hash:fetch key opal.cred for proc 9576316929 not found
[cn2:00788] [[15051,1],2] send blocking of 47 bytes to socket 11
[cn2:00788] [[15051,1],2] blocking send complete to socket 11
[cn2:00788] [[15051,1],2] tcp_peer_complete_connect: setting read event on connection to [[15051,0],1]
[cn4:17657] [[15051,1],6] MESSAGE SEND COMPLETE TO [[15051,0],3] OF 61 BYTES ON SOCKET 11
[cn4:17657] [[15051,1],6]-[[15051,0],3] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21246] mca: base: components_register: component debruijn has no register or open function
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 69
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 69 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1217
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 1217 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 369
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 369 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1251
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1251 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 69 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 1217 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 369 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 1251 bytes from [[15051,0],2] for tag 2
[cn5:13995] [[15051,1],9] posting recv
[cn2:00788] [[15051,1],2]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00788] [[15051,1],2] RECV CONNECT ACK FROM [[15051,0],1] ON SOCKET 11
[cn2:00788] [[15051,1],2] waiting for connect ack from [[15051,0],1]
[cn2:00788] [[15051,1],2] connect ack received from [[15051,0],1]
[cn2:00788] [[15051,1],2] connect-ack recvd from [[15051,0],1]
[cn2:00788] [[15051,1],2] connect-ack header from [[15051,0],1] is okay
[cn2:00788] [[15051,1],2] waiting for connect ack from [[15051,0],1]
[cn2:00788] [[15051,1],2] connect ack received from [[15051,0],1]
[cn2:00788] [[15051,1],2] connect-ack version from [[15051,0],1] matches ours
[cn2:00788] [[15051,1],2] connect-ack [[15051,0],1] authenticated
[cn2:00788] [[15051,1],2]-[[15051,0],1] tcp_peer_connected on socket 11
[cn2:00788] [[15051,1],2]-[[15051,0],1] connected: 192.168.1.2 - 192.168.1.2 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn2:00788] [[15051,1],2]:tcp:recv:handler starting send/recv events
[cn2:00788] [[15051,1],2] tcp:set_module called for peer [[15051,0],1]
[cn2:00788] [[15051,1],2] tcp:send_handler called to send to peer [[15051,0],1]
[cn2:00788] [[15051,1],2] tcp:send_handler SENDING TO [[15051,0],1]
[cn4:17658] mca: base: components_open: opening grpcomm components
[cn4:17658] mca: base: components_open: found loaded component bad
[cn4:17658] mca: base: components_open: component bad open function successful
[cn4:17658] mca:base:select: Auto-selecting grpcomm components
[cn4:17658] mca:base:select:(grpcomm) Querying component [bad]
[cn3:21245] procdir: /tmp/openmpi-sessions-artpol@cn3_0/15051/1/4
[cn3:21245] jobdir: /tmp/openmpi-sessions-artpol@cn3_0/15051/1
[cn3:21245] top: openmpi-sessions-artpol@cn3_0
[cn3:21245] tmp: /tmp
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 18166251521
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 18166251521
[cn3:21245] [[15051,1],4] routed_binomial: init routes for proc job [15051,1]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
	daemon uri 986382336.2;tcp://192.168.1.3:49500
[cn3:21245] [[15051,1],4] [[15051,0],2] with daemon registering
[cn3:21245] [[15051,1],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn3:21245] [[15051,1],4] rml_send_buffer to peer [[15051,0],2] at tag 1
[cn3:21245] [[15051,1],4] registering sync waiting for ack
[cn3:21245] [[15051,1],4] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 543
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 543 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 275
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 275 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 481
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 481 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1141
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1141 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 543 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 275 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 481 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 1141 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: registering grpcomm components
[cn5:13994] mca: base: components_register: found loaded component bad
[cn5:13994] mca: base: components_register: component bad has no register or open function
[cn5:13994] mca: base: components_open: opening grpcomm components
[cn5:13994] mca: base: components_open: found loaded component bad
[cn5:13994] mca: base: components_open: component bad open function successful
[cn5:13994] mca:base:select: Auto-selecting grpcomm components
[cn2:00788] [[15051,1],2] MESSAGE SEND COMPLETE TO [[15051,0],1] OF 61 BYTES ON SOCKET 11
[cn2:00788] [[15051,1],2]-[[15051,0],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17658] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn4:17658] mca:base:select:(grpcomm) Selected component [bad]
[cn4:17658] [[15051,1],7] grpcomm:base:receive start comm
[cn4:17658] [[15051,1],7] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn4:17658] [[15051,1],7] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn4:17658] [[15051,1],7] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn3:21246] mca: base: components_register: found loaded component direct
[cn3:21246] mca: base: components_register: component direct has no register or open function
[cn3:21246] mca: base: components_register: found loaded component radix
[cn3:21246] mca: base: components_register: component radix register function successful
[cn3:21246] mca: base: components_open: opening routed components
[cn3:21246] mca: base: components_open: found loaded component binomial
[cn3:21246] mca: base: components_open: found loaded component debruijn
[cn3:21246] mca: base: components_open: found loaded component direct
[cn3:21246] mca: base: components_open: found loaded component radix
[cn3:21246] mca:base:select: Auto-selecting routed components
[cn3:21246] mca:base:select:(routed) Querying component [binomial]
[cn3:21246] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn3:21246] mca:base:select:(routed) Querying component [debruijn]
[cn3:21246] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn3:21246] mca:base:select:(routed) Querying component [direct]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 622
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 622 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 286
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 286 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 108
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 108 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 577
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 577 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 622 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 286 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 108 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 577 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1251
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1251 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 267
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 267 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 160
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 160 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 109
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 109 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] mca:base:select:(grpcomm) Querying component [bad]
[cn5:13994] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn5:13994] mca:base:select:(grpcomm) Selected component [bad]
[cn5:13994] [[15051,1],8] grpcomm:base:receive start comm
[cn5:13994] [[15051,1],8] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn5:13994] [[15051,1],8] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn5:13994] [[15051,1],8] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn5:13994] [[15051,1],8] setting up session dir with
	tmpdir: /tmp
	host cn5
[cn2:00789] mca: base: components_open: opening dstore components
[cn2:00789] mca: base: components_open: found loaded component hash
[cn2:00789] mca:dstore:select: checking available component hash
[cn2:00789] mca:dstore:select: Querying component [hash]
[cn4:17658] [[15051,1],7] setting up session dir with
	tmpdir: /tmp
	host cn4
[cn3:21246] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn3:21246] mca:base:select:(routed) Querying component [radix]
[cn3:21246] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn3:21246] mca:base:select:(routed) Selected component [binomial]
[cn3:21246] mca: base: close: unloading component debruijn
[cn3:21246] mca: base: close: unloading component direct
[cn3:21246] mca: base: close: unloading component radix
[cn3:21246] orte_routed_base_select: initializing selected component binomial
[cn1:13986] [[15051,0],0] message received 1251 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 267 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 160 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 109 bytes from [[15051,0],2] for tag 2
[cn5:13995] procdir: /tmp/openmpi-sessions-artpol@cn5_0/15051/1/9
[cn5:13995] jobdir: /tmp/openmpi-sessions-artpol@cn5_0/15051/1
[cn5:13995] top: openmpi-sessions-artpol@cn5_0
[cn5:13995] tmp: /tmp
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 39641088001
[cn5:13995] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 39641088001
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 39641088001
[cn5:13995] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 39641088001
[cn5:13995] [[15051,1],9] routed_binomial: init routes for proc job [15051,1]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
	daemon uri 986382336.4;tcp://192.168.1.5:46667
[cn5:13995] [[15051,1],9] [[15051,0],4] with daemon registering
[cn5:13995] [[15051,1],9] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn5:13995] [[15051,1],9] rml_send_buffer to peer [[15051,0],4] at tag 1
[cn5:13995] [[15051,1],9] registering sync waiting for ack
[cn5:13995] [[15051,1],9] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn2:00789] mca: base: components_register: registering grpcomm components
[cn2:00789] mca: base: components_register: found loaded component bad
[cn2:00789] mca: base: components_register: component bad has no register or open function
[cn4:17658] [[15051,1],7] posting recv
[cn4:17658] [[15051,1],7] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn3:21245] [[15051,1],4]: set_addr to uri 986382336.2;tcp://192.168.1.3:49500
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 419
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 419 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 738
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 738 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 289
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 289 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 221
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 221 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 419 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 738 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 289 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 221 bytes from [[15051,0],2] for tag 2
[cn5:13994] [[15051,1],8] posting recv
[cn5:13994] [[15051,1],8] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn5:13994] [[15051,1],8] posting recv
[cn5:13994] [[15051,1],8] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn5:13994] [[15051,1],8] posting recv
[cn5:13994] [[15051,1],8] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn2:00789] mca: base: components_open: opening grpcomm components
[cn2:00789] mca: base: components_open: found loaded component bad
[cn2:00789] mca: base: components_open: component bad open function successful
[cn2:00789] mca:base:select: Auto-selecting grpcomm components
[cn2:00789] mca:base:select:(grpcomm) Querying component [bad]
[cn2:00789] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn2:00789] mca:base:select:(grpcomm) Selected component [bad]
[cn2:00789] [[15051,1],3] grpcomm:base:receive start comm
[cn2:00789] [[15051,1],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn2:00789] [[15051,1],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn4:17658] [[15051,1],7] posting recv
[cn4:17658] [[15051,1],7] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn4:17658] [[15051,1],7] posting recv
[cn4:17658] [[15051,1],7] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn3:21245] [[15051,1],4]:set_addr checking if peer [[15051,0],2] is reachable via component tcp
[cn3:21245] [[15051,1],4] oob:tcp: working peer [[15051,0],2] address tcp://192.168.1.3:49500
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 121
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 121 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 190
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 190 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 434
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 434 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 757
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 757 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 121 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 190 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 434 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 757 bytes from [[15051,0],2] for tag 2
[cn5:13995] [[15051,1],9] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn2:00789] [[15051,1],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn2:00789] [[15051,1],3] setting up session dir with
	tmpdir: /tmp
	host cn2
[cn4:17658] procdir: /tmp/openmpi-sessions-artpol@cn4_0/15051/1/7
[cn4:17658] jobdir: /tmp/openmpi-sessions-artpol@cn4_0/15051/1
[cn4:17658] top: openmpi-sessions-artpol@cn4_0
[cn4:17658] tmp: /tmp
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 31051153409
[cn3:21245] [[15051,1],4] PEER [[15051,0],2] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn3:21245] [[15051,1],4] PASSING ADDR 192.168.1.3 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21245] [[15051,1],4]:tcp set addr for peer [[15051,0],2]
[cn3:21245] [[15051,1],4]: peer [[15051,0],2] is reachable via component tcp
[cn3:21245] [[15051,1],4] posting recv
[cn3:21245] [[15051,1],4] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn3:21245] [[15051,1],4] rml_send_msg to peer [[15051,0],2] at tag 1
[cn3:21245] [[15051,1],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21245] [[15051,1],4]:tcp:processing set_peer cmd for interface eth0
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 206
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 206 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 160
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 160 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 926
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 926 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1110
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1110 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 206 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 160 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 926 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 1110 bytes from [[15051,0],2] for tag 2
[cn5:13994] procdir: /tmp/openmpi-sessions-artpol@cn5_0/15051/1/8
[cn5:13994] jobdir: /tmp/openmpi-sessions-artpol@cn5_0/15051/1
[cn5:13994] top: openmpi-sessions-artpol@cn5_0
[cn2:00789] [[15051,1],3] posting recv
[cn2:00789] [[15051,1],3] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 31051153409
[cn4:17658] [[15051,1],7] routed_binomial: init routes for proc job [15051,1]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
	daemon uri 986382336.3;tcp://192.168.1.4:54546
[cn4:17658] [[15051,1],7] [[15051,0],3] with daemon registering
[cn4:17658] [[15051,1],7] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn4:17658] [[15051,1],7] rml_send_buffer to peer [[15051,0],3] at tag 1
[cn4:17658] [[15051,1],7] registering sync waiting for ack
[cn4:17658] [[15051,1],7] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn4:17658] [[15051,1],7]: set_addr to uri 986382336.3;tcp://192.168.1.4:54546
[cn3:21245] [[15051,1],4] SET_PEER ADDING PEER [[15051,0],2]
[cn3:21245] [[15051,1],4] set_peer: peer [[15051,0],2] is listening on interface eth0 net 192.168.1.3 port 49500
[cn3:21245] [[15051,1],4] oob:base:send to target [[15051,0],2]
[cn3:21245] [[15051,1],4] oob:tcp:send_nb to peer [[15051,0],2]:1
[cn3:21245] [[15051,1],4] tcp:send_nb to peer [[15051,0],2]
[cn3:21245] [[15051,1],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],2]
[cn3:21245] [[15051,1],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],2]:1
[cn3:21245] [[15051,1],4] routed_binomial_get([[15051,0],2]) --> [[15051,0],2]
[cn3:21245] [[15051,1],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],2]
[cn3:21245] [[15051,1],4] tcp:send_nb: initiating connection to [[15051,0],2]
[cn3:21245] [[15051,1],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 711
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 711 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 289
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 289 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 472
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 472 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 575
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 575 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 711 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 289 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 472 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 575 bytes from [[15051,0],2] for tag 2
[cn5:13994] tmp: /tmp
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 35346120705
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 35346120705
[cn5:13994] [[15051,1],8] routed_binomial: init routes for proc job [15051,1]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
	daemon uri 986382336.4;tcp://192.168.1.5:46667
[cn5:13994] [[15051,1],8] [[15051,0],4] with daemon registering
[cn2:00789] [[15051,1],3] posting recv
[cn2:00789] [[15051,1],3] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn2:00789] [[15051,1],3] posting recv
[cn2:00789] [[15051,1],3] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn4:17658] [[15051,1],7]:set_addr checking if peer [[15051,0],3] is reachable via component tcp
[cn4:17658] [[15051,1],7] oob:tcp: working peer [[15051,0],3] address tcp://192.168.1.4:54546
[cn4:17658] [[15051,1],7] PEER [[15051,0],3] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17658] [[15051,1],7] PASSING ADDR 192.168.1.4 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn4:17658] [[15051,1],7]:tcp set addr for peer [[15051,0],3]
[cn3:21245] [[15051,1],4] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],2] via interface eth0
[cn3:21245] [[15051,1],4] oob:tcp:peer creating socket to [[15051,0],2]
[cn3:21245] [[15051,1],4] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],2] via interface eth0 on socket 11
[cn3:21245] [[15051,1],4] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],2] on 192.168.1.3:49500 - 0 retries
[cn3:21245] [[15051,1],4] waiting for connect completion to [[15051,0],2] - activating send event
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 766
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 766 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 206
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 206 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1452
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 1452 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 810
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 810 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 766 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 206 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 1452 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 810 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1307
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1307 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] [[15051,1],8] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn5:13994] [[15051,1],8] rml_send_buffer to peer [[15051,0],4] at tag 1
[cn5:13994] [[15051,1],8] registering sync waiting for ack
[cn5:13994] [[15051,1],8] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn5:13994] [[15051,1],8]: set_addr to uri 986382336.4;tcp://192.168.1.5:46667
[cn5:13994] [[15051,1],8]:set_addr checking if peer [[15051,0],4] is reachable via component tcp
[cn5:13994] [[15051,1],8] oob:tcp: working peer [[15051,0],4] address tcp://192.168.1.5:46667
[cn5:13994] [[15051,1],8] PEER [[15051,0],4] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn2:00789] procdir: /tmp/openmpi-sessions-artpol@cn2_0/15051/1/3
[cn2:00789] jobdir: /tmp/openmpi-sessions-artpol@cn2_0/15051/1
[cn2:00789] top: openmpi-sessions-artpol@cn2_0
[cn4:17658] [[15051,1],7]: peer [[15051,0],3] is reachable via component tcp
[cn4:17658] [[15051,1],7] posting recv
[cn4:17658] [[15051,1],7] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn4:17658] [[15051,1],7] rml_send_msg to peer [[15051,0],3] at tag 1
[cn4:17658] [[15051,1],7] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17658] [[15051,1],7]:tcp:processing set_peer cmd for interface eth0
[cn4:17658] [[15051,1],7] SET_PEER ADDING PEER [[15051,0],3]
[cn4:17658] [[15051,1],7] set_peer: peer [[15051,0],3] is listening on interface eth0 net 192.168.1.4 port 54546
[cn4:17658] [[15051,1],7] oob:base:send to target [[15051,0],3]
[cn4:17658] [[15051,1],7] oob:tcp:send_nb to peer [[15051,0],3]:1
[cn4:17658] [[15051,1],7] tcp:send_nb to peer [[15051,0],3]
[cn4:17658] [[15051,1],7]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],3]
[cn4:17658] [[15051,1],7]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],3]:1
[cn4:17658] [[15051,1],7] routed_binomial_get([[15051,0],3]) --> [[15051,0],3]
[cn4:17658] [[15051,1],7]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],3]
[cn4:17658] [[15051,1],7] tcp:send_nb: initiating connection to [[15051,0],3]
[cn3:21245] [[15051,1],4] tcp:send_handler called to send to peer [[15051,0],2]
[cn3:21245] [[15051,1],4] tcp:send_handler CONNECTING
[cn3:21245] [[15051,1],4]:tcp:complete_connect called for peer [[15051,0],2] on socket 11
[cn3:21245] [[15051,1],4] tcp_peer_complete_connect: sending ack to [[15051,0],2]
[cn3:21245] [[15051,1],4] SEND CONNECT ACK
[cn3:21245] fetching data from INTERNAL dstore
[cn3:21245] dstore:hash:fetch: searching for key opal.cred on proc 18166251521
[cn3:21245] dstore_hash:fetch key opal.cred for proc 18166251521 not found
[cn3:21245] [[15051,1],4] send blocking of 47 bytes to socket 11
[cn3:21245] [[15051,1],4] blocking send complete to socket 11
[cn3:21245] [[15051,1],4] tcp_peer_complete_connect: setting read event on connection to [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1019
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1019 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 711
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 711 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 709
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 709 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 1307 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] message received 1019 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 711 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 709 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1291
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1291 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 394
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 394 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 810
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 810 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 185
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 185 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21245] [[15051,1],4]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21245] [[15051,1],4] RECV CONNECT ACK FROM [[15051,0],2] ON SOCKET 11
[cn3:21245] [[15051,1],4] waiting for connect ack from [[15051,0],2]
[cn3:21245] [[15051,1],4] connect ack received from [[15051,0],2]
[cn3:21245] [[15051,1],4] connect-ack recvd from [[15051,0],2]
[cn3:21245] [[15051,1],4] connect-ack header from [[15051,0],2] is okay
[cn3:21245] [[15051,1],4] waiting for connect ack from [[15051,0],2]
[cn3:21245] [[15051,1],4] connect ack received from [[15051,0],2]
[cn3:21245] [[15051,1],4] connect-ack version from [[15051,0],2] matches ours
[cn3:21245] [[15051,1],4] connect-ack [[15051,0],2] authenticated
[cn3:21245] [[15051,1],4]-[[15051,0],2] tcp_peer_connected on socket 11
[cn3:21245] [[15051,1],4]-[[15051,0],2] connected: 192.168.1.3 - 192.168.1.3 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn3:21245] [[15051,1],4]:tcp:recv:handler starting send/recv events
[cn3:21245] [[15051,1],4] tcp:set_module called for peer [[15051,0],2]
[cn3:21245] [[15051,1],4] tcp:send_handler called to send to peer [[15051,0],2]
[cn3:21245] [[15051,1],4] tcp:send_handler SENDING TO [[15051,0],2]
[cn3:21245] [[15051,1],4] MESSAGE SEND COMPLETE TO [[15051,0],2] OF 61 BYTES ON SOCKET 11
[cn5:13994] [[15051,1],8] PASSING ADDR 192.168.1.5 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn5:13994] [[15051,1],8]:tcp set addr for peer [[15051,0],4]
[cn5:13994] [[15051,1],8]: peer [[15051,0],4] is reachable via component tcp
[cn5:13994] [[15051,1],8] posting recv
[cn5:13994] [[15051,1],8] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn5:13994] [[15051,1],8] rml_send_msg to peer [[15051,0],4] at tag 1
[cn5:13994] [[15051,1],8] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13994] [[15051,1],8]:tcp:processing set_peer cmd for interface eth0
[cn5:13994] [[15051,1],8] SET_PEER ADDING PEER [[15051,0],4]
[cn5:13994] [[15051,1],8] set_peer: peer [[15051,0],4] is listening on interface eth0 net 192.168.1.5 port 46667
[cn5:13994] [[15051,1],8] oob:base:send to target [[15051,0],4]
[cn5:13994] [[15051,1],8] oob:tcp:send_nb to peer [[15051,0],4]:1
[cn5:13994] [[15051,1],8] tcp:send_nb to peer [[15051,0],4]
[cn2:00789] tmp: /tmp
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 13871284225
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 13871284225
[cn2:00789] [[15051,1],3] routed_binomial: init routes for proc job [15051,1]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
	daemon uri 986382336.1;tcp://192.168.1.2:41722
[cn2:00789] [[15051,1],3] [[15051,0],1] with daemon registering
[cn4:17658] [[15051,1],7]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],3]
[cn4:17658] [[15051,1],7] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],3] via interface eth0
[cn4:17658] [[15051,1],7] oob:tcp:peer creating socket to [[15051,0],3]
[cn4:17658] [[15051,1],7] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],3] via interface eth0 on socket 11
[cn4:17658] [[15051,1],7] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],3] on 192.168.1.4:54546 - 0 retries
[cn4:17658] [[15051,1],7] waiting for connect completion to [[15051,0],3] - activating send event
[cn1:14002] mca: base: components_register: registering oob components
[cn1:14002] mca: base: components_register: found loaded component tcp
[cn1:14002] mca: base: components_register: component tcp register function successful
[cn1:14002] mca: base: components_open: opening oob components
[cn1:14002] mca: base: components_open: found loaded component tcp
[cn1:14002] mca: base: components_open: component tcp open function successful
[cn1:14002] mca:oob:select: checking available component tcp
[cn1:14002] mca:oob:select: Querying component [tcp]
[cn1:14002] oob:tcp: component_available called
[cn1:13986] [[15051,0],0] message received 1291 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 394 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 810 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 185 bytes from [[15051,0],2] for tag 2
[cn5:13994] [[15051,1],8]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],4]
[cn5:13994] [[15051,1],8]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],4]:1
[cn5:13994] [[15051,1],8] routed_binomial_get([[15051,0],4]) --> [[15051,0],4]
[cn5:13994] [[15051,1],8]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],4]
[cn5:13994] [[15051,1],8] tcp:send_nb: initiating connection to [[15051,0],4]
[cn5:13994] [[15051,1],8]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],4]
[cn5:13994] [[15051,1],8] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],4] via interface eth0
[cn5:13994] [[15051,1],8] oob:tcp:peer creating socket to [[15051,0],4]
[cn5:13994] [[15051,1],8] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],4] via interface eth0 on socket 11
[cn5:13994] [[15051,1],8] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],4] on 192.168.1.5:46667 - 0 retries
[cn5:13994] [[15051,1],8] waiting for connect completion to [[15051,0],4] - activating send event
[cn2:00789] [[15051,1],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn2:00789] [[15051,1],3] rml_send_buffer to peer [[15051,0],1] at tag 1
[cn2:00789] [[15051,1],3] registering sync waiting for ack
[cn2:00789] [[15051,1],3] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn4:17658] [[15051,1],7] tcp:send_handler called to send to peer [[15051,0],3]
[cn4:17658] [[15051,1],7] tcp:send_handler CONNECTING
[cn4:17658] [[15051,1],7]:tcp:complete_connect called for peer [[15051,0],3] on socket 11
[cn4:17658] [[15051,1],7] tcp_peer_complete_connect: sending ack to [[15051,0],3]
[cn4:17658] [[15051,1],7] SEND CONNECT ACK
[cn4:17658] fetching data from INTERNAL dstore
[cn4:17658] dstore:hash:fetch: searching for key opal.cred on proc 31051153409
[cn4:17658] dstore_hash:fetch key opal.cred for proc 31051153409 not found
[cn4:17658] [[15051,1],7] send blocking of 47 bytes to socket 11
[cn4:17658] [[15051,1],7] blocking send complete to socket 11
[cn4:17658] [[15051,1],7] tcp_peer_complete_connect: setting read event on connection to [[15051,0],3]
[cn3:21245] [[15051,1],4]-[[15051,0],2] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 645
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 645 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 206
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 206 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 103
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 103 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 176
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 176 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 645 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 206 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 103 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 176 bytes from [[15051,0],2] for tag 2
[cn5:13994] [[15051,1],8] tcp:send_handler called to send to peer [[15051,0],4]
[cn5:13994] [[15051,1],8] tcp:send_handler CONNECTING
[cn5:13994] [[15051,1],8]:tcp:complete_connect called for peer [[15051,0],4] on socket 11
[cn5:13994] [[15051,1],8] tcp_peer_complete_connect: sending ack to [[15051,0],4]
[cn5:13994] [[15051,1],8] SEND CONNECT ACK
[cn5:13994] fetching data from INTERNAL dstore
[cn5:13994] dstore:hash:fetch: searching for key opal.cred on proc 35346120705
[cn5:13994] dstore_hash:fetch key opal.cred for proc 35346120705 not found
[cn5:13994] [[15051,1],8] send blocking of 47 bytes to socket 11
[cn2:00789] [[15051,1],3]: set_addr to uri 986382336.1;tcp://192.168.1.2:41722
[cn2:00789] [[15051,1],3]:set_addr checking if peer [[15051,0],1] is reachable via component tcp
[cn4:17658] [[15051,1],7]:tcp:recv:handler called for peer [[15051,0],3]
[cn3:21246] mca: base: components_register: registering dstore components
[cn3:21246] mca: base: components_register: found loaded component hash
[cn1:14002] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn1:14002] [[15051,1],1] oob:tcp:init rejecting loopback interface lo
[cn1:14002] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn1:14002] [[15051,1],1] oob:tcp:init creating module for V4 address on interface eth0
[cn1:14002] [[15051,1],1] creating OOB-TCP module for interface eth0
[cn1:14002] [[15051,1],1] oob:tcp:init adding 192.168.1.1 to our list of V4 connections
[cn1:14002] WORKING INTERFACE 3 KERNEL INDEX 3 FAMILY: V4
[cn1:14002] [[15051,1],1] oob:tcp:init creating module for V4 address on interface eth1
[cn1:14002] [[15051,1],1] creating OOB-TCP module for interface eth1
[cn1:14002] [[15051,1],1] oob:tcp:init adding 192.168.10.2 to our list of V4 connections
[cn1:14002] [[15051,1],1] TCP STARTUP
[cn1:14002] [[15051,1],1] attempting to bind to IPv4 port 0
[cn1:14002] [[15051,1],1] assigned IPv4 port 53609
[cn1:14002] mca:oob:select: Adding component to end
[cn1:14002] mca:oob:select: Found 1 active transports
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 195
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 195 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 851
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 851 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 925
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 925 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 374
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 374 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 195 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 851 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 925 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 374 bytes from [[15051,0],2] for tag 2
[cn5:13994] [[15051,1],8] blocking send complete to socket 11
[cn5:13994] [[15051,1],8] tcp_peer_complete_connect: setting read event on connection to [[15051,0],4]
[cn2:00789] [[15051,1],3] oob:tcp: working peer [[15051,0],1] address tcp://192.168.1.2:41722
[cn2:00789] [[15051,1],3] PEER [[15051,0],1] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn2:00789] [[15051,1],3] PASSING ADDR 192.168.1.2 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00789] [[15051,1],3]:tcp set addr for peer [[15051,0],1]
[cn2:00789] [[15051,1],3]: peer [[15051,0],1] is reachable via component tcp
[cn2:00789] [[15051,1],3] posting recv
[cn2:00789] [[15051,1],3] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn2:00789] [[15051,1],3] rml_send_msg to peer [[15051,0],1] at tag 1
[cn2:00789] [[15051,1],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00789] [[15051,1],3]:tcp:processing set_peer cmd for interface eth0
[cn4:17658] [[15051,1],7] RECV CONNECT ACK FROM [[15051,0],3] ON SOCKET 11
[cn4:17658] [[15051,1],7] waiting for connect ack from [[15051,0],3]
[cn4:17658] [[15051,1],7] connect ack received from [[15051,0],3]
[cn4:17658] [[15051,1],7] connect-ack recvd from [[15051,0],3]
[cn4:17658] [[15051,1],7] connect-ack header from [[15051,0],3] is okay
[cn4:17658] [[15051,1],7] waiting for connect ack from [[15051,0],3]
[cn4:17658] [[15051,1],7] connect ack received from [[15051,0],3]
[cn4:17658] [[15051,1],7] connect-ack version from [[15051,0],3] matches ours
[cn4:17658] [[15051,1],7] connect-ack [[15051,0],3] authenticated
[cn4:17658] [[15051,1],7]-[[15051,0],3] tcp_peer_connected on socket 11
[cn4:17658] [[15051,1],7]-[[15051,0],3] connected: 192.168.1.4 - 192.168.1.4 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn4:17658] [[15051,1],7]:tcp:recv:handler starting send/recv events
[cn3:21246] mca: base: components_register: component hash register function successful
[cn3:21246] mca: base: components_open: opening dstore components
[cn3:21246] mca: base: components_open: found loaded component hash
[cn3:21246] mca:dstore:select: checking available component hash
[cn3:21246] mca:dstore:select: Querying component [hash]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1217
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1217 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 679
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 679 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 494
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 494 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 267
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 267 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 1217 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 679 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 494 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 267 bytes from [[15051,0],2] for tag 2
[cn5:13994] [[15051,1],8]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13994] [[15051,1],8] RECV CONNECT ACK FROM [[15051,0],4] ON SOCKET 11
[cn5:13994] [[15051,1],8] waiting for connect ack from [[15051,0],4]
[cn5:13994] [[15051,1],8] connect ack received from [[15051,0],4]
[cn5:13994] [[15051,1],8] connect-ack recvd from [[15051,0],4]
[cn5:13994] [[15051,1],8] connect-ack header from [[15051,0],4] is okay
[cn5:13994] [[15051,1],8] waiting for connect ack from [[15051,0],4]
[cn5:13994] [[15051,1],8] connect ack received from [[15051,0],4]
[cn5:13994] [[15051,1],8] connect-ack version from [[15051,0],4] matches ours
[cn5:13994] [[15051,1],8] connect-ack [[15051,0],4] authenticated
[cn5:13994] [[15051,1],8]-[[15051,0],4] tcp_peer_connected on socket 11
[cn5:13994] [[15051,1],8]-[[15051,0],4] connected: 192.168.1.5 - 192.168.1.5 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn5:13994] [[15051,1],8]:tcp:recv:handler starting send/recv events
[cn5:13994] [[15051,1],8] tcp:set_module called for peer [[15051,0],4]
[cn5:13994] [[15051,1],8] tcp:send_handler called to send to peer [[15051,0],4]
[cn5:13994] [[15051,1],8] tcp:send_handler SENDING TO [[15051,0],4]
[cn2:00789] [[15051,1],3] SET_PEER ADDING PEER [[15051,0],1]
[cn2:00789] [[15051,1],3] set_peer: peer [[15051,0],1] is listening on interface eth0 net 192.168.1.2 port 41722
[cn2:00789] [[15051,1],3] oob:base:send to target [[15051,0],1]
[cn2:00789] [[15051,1],3] oob:tcp:send_nb to peer [[15051,0],1]:1
[cn2:00789] [[15051,1],3] tcp:send_nb to peer [[15051,0],1]
[cn2:00789] [[15051,1],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],1]
[cn2:00789] [[15051,1],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],1]:1
[cn4:17658] [[15051,1],7] tcp:set_module called for peer [[15051,0],3]
[cn4:17658] [[15051,1],7] tcp:send_handler called to send to peer [[15051,0],3]
[cn4:17658] [[15051,1],7] tcp:send_handler SENDING TO [[15051,0],3]
[cn4:17658] [[15051,1],7] MESSAGE SEND COMPLETE TO [[15051,0],3] OF 61 BYTES ON SOCKET 11
[cn4:17658] [[15051,1],7]-[[15051,0],3] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21246] mca: base: components_register: registering grpcomm components
[cn3:21246] mca: base: components_register: found loaded component bad
[cn3:21246] mca: base: components_register: component bad has no register or open function
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 275
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 275 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1006
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 1006 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 273
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 273 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 738
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 738 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 275 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 1006 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 273 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 738 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 199
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 199 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 810
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 810 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 108
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 108 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 190
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 190 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] [[15051,1],8] MESSAGE SEND COMPLETE TO [[15051,0],4] OF 61 BYTES ON SOCKET 11
[cn5:13994] [[15051,1],8]-[[15051,0],4] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00789] [[15051,1],3] routed_binomial_get([[15051,0],1]) --> [[15051,0],1]
[cn2:00789] [[15051,1],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],1]
[cn2:00789] [[15051,1],3] tcp:send_nb: initiating connection to [[15051,0],1]
[cn2:00789] [[15051,1],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],1]
[cn2:00789] [[15051,1],3] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],1] via interface eth0
[cn2:00789] [[15051,1],3] oob:tcp:peer creating socket to [[15051,0],1]
[cn2:00789] [[15051,1],3] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],1] via interface eth0 on socket 11
[cn2:00789] [[15051,1],3] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],1] on 192.168.1.2:41722 - 0 retries
[cn2:00789] [[15051,1],3] waiting for connect completion to [[15051,0],1] - activating send event
[cn4:17657] [[15051,1],6]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17657] [[15051,1],6]:tcp:recv:handler CONNECTED
[cn4:17657] [[15051,1],6]:tcp:recv:handler allocate new recv msg
[cn4:17657] [[15051,1],6]:tcp:recv:handler read hdr
[cn3:21246] mca: base: components_open: opening grpcomm components
[cn3:21246] mca: base: components_open: found loaded component bad
[cn3:21246] mca: base: components_open: component bad open function successful
[cn3:21246] mca:base:select: Auto-selecting grpcomm components
[cn3:21246] mca:base:select:(grpcomm) Querying component [bad]
[cn3:21246] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn3:21246] mca:base:select:(grpcomm) Selected component [bad]
[cn3:21246] [[15051,1],5] grpcomm:base:receive start comm
[cn3:21246] [[15051,1],5] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn3:21246] [[15051,1],5] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn1:14002] mca: base: components_register: registering rml components
[cn1:14002] mca: base: components_register: found loaded component oob
[cn1:14002] mca: base: components_register: component oob has no register or open function
[cn1:14002] mca: base: components_open: opening rml components
[cn1:14002] mca: base: components_open: found loaded component oob
[cn1:14002] mca: base: components_open: component oob open function successful
[cn1:14002] orte_rml_base_select: initializing rml component oob
[cn1:13986] [[15051,0],0] message received 199 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 810 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 108 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 190 bytes from [[15051,0],2] for tag 2
[cn5:13995] [[15051,1],9] posting recv
[cn5:13995] [[15051,1],9] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn5:13995] [[15051,1],9] posting recv
[cn2:00789] [[15051,1],3] tcp:send_handler called to send to peer [[15051,0],1]
[cn2:00789] [[15051,1],3] tcp:send_handler CONNECTING
[cn2:00789] [[15051,1],3]:tcp:complete_connect called for peer [[15051,0],1] on socket 11
[cn2:00789] [[15051,1],3] tcp_peer_complete_connect: sending ack to [[15051,0],1]
[cn2:00789] [[15051,1],3] SEND CONNECT ACK
[cn2:00789] fetching data from INTERNAL dstore
[cn2:00789] dstore:hash:fetch: searching for key opal.cred on proc 13871284225
[cn2:00789] dstore_hash:fetch key opal.cred for proc 13871284225 not found
[cn2:00789] [[15051,1],3] send blocking of 47 bytes to socket 11
[cn2:00789] [[15051,1],3] blocking send complete to socket 11
[cn2:00789] [[15051,1],3] tcp_peer_complete_connect: setting read event on connection to [[15051,0],1]
[cn4:17657] [[15051,1],6]:tcp:recv:handler allocate data region of size 14140
[cn3:21246] [[15051,1],5] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn3:21246] [[15051,1],5] setting up session dir with
	tmpdir: /tmp
	host cn3
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 579
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 579 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1462
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 1462 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 179
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 179 BYTES FOR DEST [[15051,0],0] TAG 5
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 160
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 160 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 579 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 1462 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 179 bytes from [[15051,0],3] for tag 5
[cn1:13986] [[15051,0],0] plm:base:receive processing msg
[cn1:13986] [[15051,0],0] routed_binomial: init routes for HNP job [15051,1]
[cn1:13986] [[15051,0],0] routed_binomial:callback got uri 986382337.6;tcp://192.168.1.4:50185 for job [15051,1] rank 6
[cn1:13986] [[15051,0],0] routed_binomial:callback got uri 986382337.7;tcp://192.168.1.4:39807 for job [15051,1] rank 7
[cn1:13986] [[15051,0],0] plm:base:receive done processing commands
[cn1:13986] [[15051,0],0] message received 160 bytes from [[15051,0],2] for tag 2
[cn5:13995] [[15051,1],9] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn5:13995] [[15051,1],9]: set_addr to uri 986382336.4;tcp://192.168.1.5:46667
[cn5:13995] [[15051,1],9]:set_addr checking if peer [[15051,0],4] is reachable via component tcp
[cn5:13995] [[15051,1],9] oob:tcp: working peer [[15051,0],4] address tcp://192.168.1.5:46667
[cn5:13995] [[15051,1],9] PEER [[15051,0],4] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:13995] [[15051,1],9] PASSING ADDR 192.168.1.5 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00789] [[15051,1],3]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00789] [[15051,1],3] RECV CONNECT ACK FROM [[15051,0],1] ON SOCKET 11
[cn2:00789] [[15051,1],3] waiting for connect ack from [[15051,0],1]
[cn2:00789] [[15051,1],3] connect ack received from [[15051,0],1]
[cn2:00789] [[15051,1],3] connect-ack recvd from [[15051,0],1]
[cn2:00789] [[15051,1],3] connect-ack header from [[15051,0],1] is okay
[cn2:00789] [[15051,1],3] waiting for connect ack from [[15051,0],1]
[cn2:00789] [[15051,1],3] connect ack received from [[15051,0],1]
[cn2:00789] [[15051,1],3] connect-ack version from [[15051,0],1] matches ours
[cn2:00789] [[15051,1],3] connect-ack [[15051,0],1] authenticated
[cn2:00789] [[15051,1],3]-[[15051,0],1] tcp_peer_connected on socket 11
[cn2:00789] [[15051,1],3]-[[15051,0],1] connected: 192.168.1.2 - 192.168.1.2 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn2:00789] [[15051,1],3]:tcp:recv:handler starting send/recv events
[cn2:00789] [[15051,1],3] tcp:set_module called for peer [[15051,0],1]
[cn2:00789] [[15051,1],3] tcp:send_handler called to send to peer [[15051,0],1]
[cn2:00789] [[15051,1],3] tcp:send_handler SENDING TO [[15051,0],1]
[cn2:00789] [[15051,1],3] MESSAGE SEND COMPLETE TO [[15051,0],1] OF 61 BYTES ON SOCKET 11
[cn2:00789] [[15051,1],3]-[[15051,0],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21246] [[15051,1],5] posting recv
[cn3:21246] [[15051,1],5] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 932
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 932 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 273
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 273 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 351
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 351 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 289
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 289 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 932 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 273 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 351 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 289 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1291
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1291 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 108
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 108 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 555
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 555 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 206
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 206 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] [[15051,1],9]:tcp set addr for peer [[15051,0],4]
[cn5:13995] [[15051,1],9]: peer [[15051,0],4] is reachable via component tcp
[cn5:13995] [[15051,1],9] posting recv
[cn5:13995] [[15051,1],9] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn5:13995] [[15051,1],9] rml_send_msg to peer [[15051,0],4] at tag 1
[cn5:13995] [[15051,1],9] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13995] [[15051,1],9]:tcp:processing set_peer cmd for interface eth0
[cn5:13995] [[15051,1],9] SET_PEER ADDING PEER [[15051,0],4]
[cn5:13995] [[15051,1],9] set_peer: peer [[15051,0],4] is listening on interface eth0 net 192.168.1.5 port 46667
[cn5:13995] [[15051,1],9] oob:base:send to target [[15051,0],4]
[cn5:13995] [[15051,1],9] oob:tcp:send_nb to peer [[15051,0],4]:1
[cn5:13995] [[15051,1],9] tcp:send_nb to peer [[15051,0],4]
[cn2:00788] [[15051,1],2]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00788] [[15051,1],2]:tcp:recv:handler CONNECTED
[cn2:00788] [[15051,1],2]:tcp:recv:handler allocate new recv msg
[cn2:00788] [[15051,1],2]:tcp:recv:handler read hdr
[cn4:17658] [[15051,1],7]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17658] [[15051,1],7]:tcp:recv:handler CONNECTED
[cn4:17658] [[15051,1],7]:tcp:recv:handler allocate new recv msg
[cn4:17658] [[15051,1],7]:tcp:recv:handler read hdr
[cn4:17658] [[15051,1],7]:tcp:recv:handler allocate data region of size 14140
[cn3:21246] [[15051,1],5] posting recv
[cn3:21246] [[15051,1],5] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn3:21246] [[15051,1],5] posting recv
[cn3:21246] [[15051,1],5] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0] message received 1291 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 108 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 555 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 206 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 810
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 810 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 179
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 179 BYTES FOR DEST [[15051,0],0] TAG 5
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 83 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 711
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 711 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] [[15051,1],9]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],4]
[cn5:13995] [[15051,1],9]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],4]:1
[cn5:13995] [[15051,1],9] routed_binomial_get([[15051,0],4]) --> [[15051,0],4]
[cn5:13995] [[15051,1],9]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],4]
[cn5:13995] [[15051,1],9] tcp:send_nb: initiating connection to [[15051,0],4]
[cn5:13995] [[15051,1],9]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],4]
[cn5:13995] [[15051,1],9] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],4] via interface eth0
[cn5:13995] [[15051,1],9] oob:tcp:peer creating socket to [[15051,0],4]
[cn5:13995] [[15051,1],9] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],4] via interface eth0 on socket 11
[cn5:13995] [[15051,1],9] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],4] on 192.168.1.5:46667 - 0 retries
[cn5:13995] [[15051,1],9] waiting for connect completion to [[15051,0],4] - activating send event
[cn2:00788] [[15051,1],2]:tcp:recv:handler allocate data region of size 14140
[cn4:17657] [[15051,1],6]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17657] [[15051,1],6]:tcp:recv:handler CONNECTED
[cn4:17657] [[15051,1],6] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 14140 BYTES FOR DEST [[15051,1],6] TAG 20
[cn4:17657] [[15051,1],6] DELIVERING TO RML
[cn4:17657] [[15051,1],6] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17657] [[15051,1],6] message received 14140 bytes from [[15051,0],3] for tag 20
[cn3:21246] procdir: /tmp/openmpi-sessions-artpol@cn3_0/15051/1/5
[cn3:21246] jobdir: /tmp/openmpi-sessions-artpol@cn3_0/15051/1
[cn3:21246] top: openmpi-sessions-artpol@cn3_0
[cn1:13986] [[15051,0],0] message received 810 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 179 bytes from [[15051,0],1] for tag 5
[cn1:13986] [[15051,0],0] plm:base:receive processing msg
[cn1:13986] [[15051,0],0] routed_binomial: init routes for HNP job [15051,1]
[cn1:13986] [[15051,0],0] routed_binomial:callback got uri 986382337.2;tcp://192.168.1.2:37167 for job [15051,1] rank 2
[cn1:13986] [[15051,0],0] routed_binomial:callback got uri 986382337.3;tcp://192.168.1.2:34141 for job [15051,1] rank 3
[cn1:13986] [[15051,0],0] plm:base:receive done processing commands
[cn1:13986] [[15051,0],0] message received 83 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 711 bytes from [[15051,0],2] for tag 2
[cn5:13995] [[15051,1],9] tcp:send_handler called to send to peer [[15051,0],4]
[cn5:13995] [[15051,1],9] tcp:send_handler CONNECTING
[cn5:13995] [[15051,1],9]:tcp:complete_connect called for peer [[15051,0],4] on socket 11
[cn5:13995] [[15051,1],9] tcp_peer_complete_connect: sending ack to [[15051,0],4]
[cn5:13995] [[15051,1],9] SEND CONNECT ACK
[cn5:13995] fetching data from INTERNAL dstore
[cn5:13995] dstore:hash:fetch: searching for key opal.cred on proc 39641088001
[cn5:13995] dstore_hash:fetch key opal.cred for proc 39641088001 not found
[cn5:13995] [[15051,1],9] send blocking of 47 bytes to socket 11
[cn5:13995] [[15051,1],9] blocking send complete to socket 11
[cn5:13995] [[15051,1],9] tcp_peer_complete_connect: setting read event on connection to [[15051,0],4]
[cn4:17657] [[15051,1],6] registering sync ack recvd
[cn3:21246] tmp: /tmp
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 22461218817
[cn3:21246] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 22461218817
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 22461218817
[cn3:21246] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 22461218817
[cn3:21246] [[15051,1],5] routed_binomial: init routes for proc job [15051,1]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
	daemon uri 986382336.2;tcp://192.168.1.3:49500
[cn3:21246] [[15051,1],5] [[15051,0],2] with daemon registering
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1462
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1462 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 351
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 351 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 555
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 555 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 664
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 664 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 1462 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 351 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 555 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 664 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 555
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 555 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 83 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1121
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1121 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] [[15051,1],9]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13995] [[15051,1],9] RECV CONNECT ACK FROM [[15051,0],4] ON SOCKET 11
[cn5:13995] [[15051,1],9] waiting for connect ack from [[15051,0],4]
[cn5:13995] [[15051,1],9] connect ack received from [[15051,0],4]
[cn5:13995] [[15051,1],9] connect-ack recvd from [[15051,0],4]
[cn5:13995] [[15051,1],9] connect-ack header from [[15051,0],4] is okay
[cn5:13995] [[15051,1],9] waiting for connect ack from [[15051,0],4]
[cn5:13995] [[15051,1],9] connect ack received from [[15051,0],4]
[cn5:13995] [[15051,1],9] connect-ack version from [[15051,0],4] matches ours
[cn5:13995] [[15051,1],9] connect-ack [[15051,0],4] authenticated
[cn5:13995] [[15051,1],9]-[[15051,0],4] tcp_peer_connected on socket 11
[cn5:13995] [[15051,1],9]-[[15051,0],4] connected: 192.168.1.5 - 192.168.1.5 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn5:13995] [[15051,1],9]:tcp:recv:handler starting send/recv events
[cn5:13995] [[15051,1],9] tcp:set_module called for peer [[15051,0],4]
[cn5:13995] [[15051,1],9] tcp:send_handler called to send to peer [[15051,0],4]
[cn5:13995] [[15051,1],9] tcp:send_handler SENDING TO [[15051,0],4]
[cn5:13995] [[15051,1],9] MESSAGE SEND COMPLETE TO [[15051,0],4] OF 61 BYTES ON SOCKET 11
[cn5:13995] [[15051,1],9]-[[15051,0],4] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00789] [[15051,1],3]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00789] [[15051,1],3]:tcp:recv:handler CONNECTED
[cn2:00789] [[15051,1],3]:tcp:recv:handler allocate new recv msg
[cn2:00789] [[15051,1],3]:tcp:recv:handler read hdr
[cn2:00789] [[15051,1],3]:tcp:recv:handler allocate data region of size 14140
[cn4:17658] [[15051,1],7]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17658] [[15051,1],7]:tcp:recv:handler CONNECTED
[cn4:17658] [[15051,1],7] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 14140 BYTES FOR DEST [[15051,1],7] TAG 20
[cn4:17658] [[15051,1],7] DELIVERING TO RML
[cn4:17658] [[15051,1],7] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17658] [[15051,1],7] message received 14140 bytes from [[15051,0],3] for tag 20
[cn3:21246] [[15051,1],5] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn3:21246] [[15051,1],5] rml_send_buffer to peer [[15051,0],2] at tag 1
[cn3:21246] [[15051,1],5] registering sync waiting for ack
[cn3:21246] [[15051,1],5] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn3:21246] [[15051,1],5]: set_addr to uri 986382336.2;tcp://192.168.1.3:49500
[cn3:21246] [[15051,1],5]:set_addr checking if peer [[15051,0],2] is reachable via component tcp
[cn3:21246] [[15051,1],5] oob:tcp: working peer [[15051,0],2] address tcp://192.168.1.3:49500
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 273
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 273 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 555 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 83 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 1121 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] message received 273 bytes from [[15051,0],4] for tag 2
[cn2:00788] [[15051,1],2]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00788] [[15051,1],2]:tcp:recv:handler CONNECTED
[cn2:00788] [[15051,1],2] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 14140 BYTES FOR DEST [[15051,1],2] TAG 20
[cn2:00788] [[15051,1],2] DELIVERING TO RML
[cn2:00788] [[15051,1],2] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00788] [[15051,1],2] message received 14140 bytes from [[15051,0],1] for tag 20
[cn4:17658] [[15051,1],7] registering sync ack recvd
[cn3:21246] [[15051,1],5] PEER [[15051,0],2] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn3:21246] [[15051,1],5] PASSING ADDR 192.168.1.3 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21246] [[15051,1],5]:tcp set addr for peer [[15051,0],2]
[cn3:21246] [[15051,1],5]: peer [[15051,0],2] is reachable via component tcp
[cn3:21246] [[15051,1],5] posting recv
[cn3:21246] [[15051,1],5] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn3:21246] [[15051,1],5] rml_send_msg to peer [[15051,0],2] at tag 1
[cn3:21246] [[15051,1],5] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21246] [[15051,1],5]:tcp:processing set_peer cmd for interface eth0
[cn3:21246] [[15051,1],5] SET_PEER ADDING PEER [[15051,0],2]
[cn3:21246] [[15051,1],5] set_peer: peer [[15051,0],2] is listening on interface eth0 net 192.168.1.3 port 49500
[cn3:21246] [[15051,1],5] oob:base:send to target [[15051,0],2]
[cn3:21246] [[15051,1],5] oob:tcp:send_nb to peer [[15051,0],2]:1
[cn3:21246] [[15051,1],5] tcp:send_nb to peer [[15051,0],2]
[cn5:13994] [[15051,1],8]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13994] [[15051,1],8]:tcp:recv:handler CONNECTED
[cn5:13994] [[15051,1],8]:tcp:recv:handler allocate new recv msg
[cn5:13994] [[15051,1],8]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 108
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 108 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 83 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 934
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 934 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 108 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 83 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 934 bytes from [[15051,0],2] for tag 2
[cn5:13994] [[15051,1],8]:tcp:recv:handler allocate data region of size 14140
[cn2:00788] [[15051,1],2] registering sync ack recvd
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 986382336
[cn4:17658] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 986382336
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17658] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 5281349632
[cn4:17658] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349632
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17658] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 9576316928
[cn4:17658] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316928
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17658] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 13871284224
[cn4:17658] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284224
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17658] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31051153409
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31051153409
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 18166251520
[cn4:17658] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251520
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17658] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 18446744070400966657
[cn4:17658] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 18446744070400966657
[cn4:17658] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 986382337
[cn4:17658] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 986382337
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 986382337
[cn4:17658] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 986382337
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 986382337
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 986382337
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 986382337
[cn4:17658] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 986382337
[cn4:17658] fetching data from INTERNAL dstore
[cn4:17658] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 986382337
[cn4:17658] dstore:hash:store: stori[cn3:21246] [[15051,1],5]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],2]
[cn3:21246] [[15051,1],5]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],2]:1
[cn3:21246] [[15051,1],5] routed_binomial_get([[15051,0],2]) --> [[15051,0],2]
[cn3:21246] [[15051,1],5]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],2]
[cn3:21246] [[15051,1],5] tcp:send_nb: initiating connection to [[15051,0],2]
[cn3:21246] [[15051,1],5]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],2]
[cn3:21246] [[15051,1],5] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],2] via interface eth0
[cn3:21246] [[15051,1],5] oob:tcp:peer creating socket to [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 179
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 179 BYTES FOR DEST [[15051,0],0] TAG 5
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1231
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 1231 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1167
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1167 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 179 bytes from [[15051,0],4] for tag 5
[cn1:13986] [[15051,0],0] plm:base:receive processing msg
[cn1:13986] [[15051,0],0] routed_binomial: init routes for HNP job [15051,1]
[cn1:13986] [[15051,0],0] routed_binomial:callback got uri 986382337.8;tcp://192.168.1.5:53909 for job [15051,1] rank 8
[cn1:13986] [[15051,0],0] routed_binomial:callback got uri 986382337.9;tcp://192.168.1.5:54485 for job [15051,1] rank 9
[cn1:13986] [[15051,0],0] plm:base:receive done processing commands
[cn1:13986] [[15051,0],0] message received 1231 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 1167 bytes from [[15051,0],2] for tag 2
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 986382336
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 986382336
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00788] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 5281349632
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349632
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00788] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316929
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9576316929
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 986382336
[cn4:17657] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 986382336
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17657] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 5281349632
[cn4:17657] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349632
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17657] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 9576316928
[cn4:17657] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316928
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17657] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 13871284224
[cn4:17657] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284224
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17657] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26756186113
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26756186113
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 18166251520
[cn4:17657] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251520
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17657] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 18446744070400966657
[cn4:17657] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 18446744070400966657
[cn4:17657] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 986382337
[cn4:17657] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 986382337
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 986382337
[cn4:17657] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 986382337
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 986382337
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 986382337
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 986382337
[cn4:17657] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 986382337
[cn4:17657] fetching data from INTERNAL dstore
[cn4:17657] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 986382337
[cn4:17657] dstore:hash:store: stori[cn3:21246] [[15051,1],5] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],2] via interface eth0 on socket 11
[cn3:21246] [[15051,1],5] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],2] on 192.168.1.3:49500 - 0 retries
[cn3:21246] [[15051,1],5] waiting for connect completion to [[15051,0],2] - activating send event
[cn3:21246] [[15051,1],5] tcp:send_handler called to send to peer [[15051,0],2]
[cn3:21246] [[15051,1],5] tcp:send_handler CONNECTING
[cn3:21246] [[15051,1],5]:tcp:complete_connect called for peer [[15051,0],2] on socket 11
[cn3:21246] [[15051,1],5] tcp_peer_complete_connect: sending ack to [[15051,0],2]
[cn3:21246] [[15051,1],5] SEND CONNECT ACK
[cn3:21246] fetching data from INTERNAL dstore
[cn3:21246] dstore:hash:fetch: searching for key opal.cred on proc 22461218817
[cn3:21246] dstore_hash:fetch key opal.cred for proc 22461218817 not found
[cn3:21246] [[15051,1],5] send blocking of 47 bytes to socket 11
[cn3:21246] [[15051,1],5] blocking send complete to socket 11
[cn3:21246] [[15051,1],5] tcp_peer_complete_connect: setting read event on connection to [[15051,0],2]
[cn1:14002] mca: base: components_register: registering routed components
[cn1:14002] mca: base: components_register: found loaded component binomial
[cn1:14002] mca: base: components_register: component binomial has no register or open function
[cn1:14002] mca: base: components_register: found loaded component debruijn
[cn1:14002] mca: base: components_register: component debruijn has no register or open function
[cn1:14002] mca: base: components_register: found loaded component direct
[cn1:14002] mca: base: components_register: component direct has no register or open function
[cn1:14002] mca: base: components_register: found loaded component radix
[cn1:14002] mca: base: components_register: component radix register function successful
[cn1:14002] mca: base: components_open: opening routed components
[cn1:14002] mca: base: components_open: found loaded component binomial
[cn1:14002] mca: base: components_open: found loaded component debruijn
[cn1:14002] mca: base: components_open: found loaded component direct
[cn1:14002] mca: base: components_open: found loaded component radix
[cn1:14002] mca:base:select: Auto-selecting routed components
[cn1:14002] mca:base:select:(routed) Querying component [binomial]
[cn1:14002] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 103
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 103 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4001
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 4001 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 799
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 799 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 103 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 4001 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 799 bytes from [[15051,0],2] for tag 2
[cn5:13995] [[15051,1],9]:tcp:recv:handler called for peer [[15051,0],4]
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316928
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316928
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00788] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 13871284224
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284224
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00788] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 18166251520
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251520
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00788] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 18446744070400966657
[cn2:00788] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 18446744070400966657
[cn2:00788] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 986382337
[cn2:00788] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 986382337
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 986382337
[cn2:00788] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 986382337
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 986382337
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 986382337
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 986382337
[cn2:00788] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 986382337
[cn2:00788] fetching data from INTERNAL dstore
[cn2:00788] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 986382337
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 986382337
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 986382337
[cn2:00788] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 986382337
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 5281349633
[cn2:00788] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5281349633
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 5281349633
[cn2:00788] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5281349633
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 5281349633
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 5281349633
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn2:00788] storing data in NONPEER dstore
ng key orte.hostname[OPAL_STRING] for proc 986382337
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 986382337
[cn4:17657] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 986382337
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 5281349633
[cn4:17657] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5281349633
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 5281349633
[cn4:17657] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5281349633
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 5281349633
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 5281349633
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 5281349633
[cn4:17657] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5281349633
[cn4:17657] fetching data from INTERNAL dstore
[cn4:17657] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 5281349633
[cn4:17657] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349633
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 5281349633
[cn4:17657] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5281349633
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 9576316929
[cn4:17657] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9576316929
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 9576316929
[cn4:17657] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9576316929
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 9576316929
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 9576316929
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 9576316929
[cn4:17657] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9576316929
[cn4:17657] fetching data from INTERNAL dstore
[cn4:17657] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 9576316929
[cn4:17657] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316929
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 9576316929
[cn4:17657] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9576316929
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 13871284225
[cn4:17657] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13871284225
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 13871284225
[cn4:17657] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13871284225
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 13871284225
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 13871284225
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn4:17657] storing data in NONP[cn3:21246] [[15051,1],5]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21246] [[15051,1],5] RECV CONNECT ACK FROM [[15051,0],2] ON SOCKET 11
[cn3:21246] [[15051,1],5] waiting for connect ack from [[15051,0],2]
[cn3:21246] [[15051,1],5] connect ack received from [[15051,0],2]
[cn3:21246] [[15051,1],5] connect-ack recvd from [[15051,0],2]
[cn3:21246] [[15051,1],5] connect-ack header from [[15051,0],2] is okay
[cn3:21246] [[15051,1],5] waiting for connect ack from [[15051,0],2]
[cn3:21246] [[15051,1],5] connect ack received from [[15051,0],2]
[cn3:21246] [[15051,1],5] connect-ack version from [[15051,0],2] matches ours
[cn3:21246] [[15051,1],5] connect-ack [[15051,0],2] authenticated
[cn3:21246] [[15051,1],5]-[[15051,0],2] tcp_peer_connected on socket 11
[cn1:14002] mca:base:select:(routed) Querying component [debruijn]
[cn1:14002] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn1:14002] mca:base:select:(routed) Querying component [direct]
[cn1:14002] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn1:14002] mca:base:select:(routed) Querying component [radix]
[cn1:14002] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn1:14002] mca:base:select:(routed) Selected component [binomial]
[cn1:14002] mca: base: close: unloading component debruijn
[cn1:14002] mca: base: close: unloading component direct
[cn1:14002] mca: base: close: unloading component radix
[cn1:14002] orte_routed_base_select: initializing selected component binomial
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 83 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1937
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 1937 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 693
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 693 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 83 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 1937 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 693 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 95
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 95 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 3034
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 3034 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 351
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 351 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] [[15051,1],9]:tcp:recv:handler CONNECTED
[cn2:00788] dstore:hash:store storing data for proc 5281349633
[cn2:00788] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5281349633
[cn2:00788] fetching data from INTERNAL dstore
[cn2:00788] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 5281349633
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349633
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 5281349633
[cn2:00788] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5281349633
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 9576316929
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9576316929
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9576316929
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9576316929
[cn2:00788] storing data in INTERNAL dstore
ng key orte.hostname[OPAL_STRING] for proc 986382337
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 986382337
[cn4:17658] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 986382337
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 5281349633
[cn4:17658] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5281349633
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 5281349633
[cn4:17658] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5281349633
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 5281349633
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 5281349633
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 5281349633
[cn4:17658] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5281349633
[cn4:17658] fetching data from INTERNAL dstore
[cn4:17658] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 5281349633
[cn4:17658] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349633
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 5281349633
[cn4:17658] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5281349633
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 9576316929
[cn4:17658] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9576316929
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 9576316929
[cn4:17658] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9576316929
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 9576316929
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 9576316929
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 9576316929
[cn4:17658] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9576316929
[cn4:17658] fetching data from INTERNAL dstore
[cn4:17658] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 9576316929
[cn4:17658] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316929
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 9576316929
[cn4:17658] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9576316929
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 13871284225
[cn4:17658] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13871284225
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 13871284225
[cn4:17658] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13871284225
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 13871284225
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 13871284225
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn4:17658] storing data in NONP[cn3:21246] [[15051,1],5]-[[15051,0],2] connected: 192.168.1.3 - 192.168.1.3 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn3:21246] [[15051,1],5]:tcp:recv:handler starting send/recv events
[cn3:21246] [[15051,1],5] tcp:set_module called for peer [[15051,0],2]
[cn3:21246] [[15051,1],5] tcp:send_handler called to send to peer [[15051,0],2]
[cn3:21246] [[15051,1],5] tcp:send_handler SENDING TO [[15051,0],2]
[cn3:21246] [[15051,1],5] MESSAGE SEND COMPLETE TO [[15051,0],2] OF 61 BYTES ON SOCKET 11
[cn3:21246] [[15051,1],5]-[[15051,0],2] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 95 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 3034 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 351 bytes from [[15051,0],2] for tag 2
[cn5:13995] [[15051,1],9]:tcp:recv:handler allocate new recv msg
[cn2:00788] dstore:hash:store storing data for proc 13871284225
[cn2:00788] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13871284225
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 13871284225
[cn2:00788] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13871284225
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 13871284225
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 13871284225
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 13871284225
[cn2:00788] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13871284225
[cn2:00788] fetching data from INTERNAL dstore
[cn2:00788] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 13871284225
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284225
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 13871284225
[cn2:00788] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13871284225
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 18166251521
[cn2:00788] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18166251521
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 18166251521
[cn2:00788] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18166251521
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 18166251521
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 18166251521
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 18166251521
[cn2:00788] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18166251521
[cn2:00788] fetching data from INTERNAL dstore
[cn2:00788] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 18166251521
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251521
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 18166251521
[cn2:00788] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18166251521
EER dstore
[cn4:17658] dstore:hash:store storing data for proc 13871284225
[cn4:17658] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13871284225
[cn4:17658] fetching data from INTERNAL dstore
[cn4:17658] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 13871284225
[cn4:17658] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284225
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 13871284225
[cn4:17658] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13871284225
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 18166251521
[cn4:17658] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18166251521
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 18166251521
[cn4:17658] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18166251521
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 18166251521
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 18166251521
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 18166251521
[cn4:17658] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18166251521
[cn4:17658] fetching data from INTERNAL dstore
[cn4:17658] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 18166251521
[cn4:17658] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251521
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 18166251521
[cn4:17658] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18166251521
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 22461218817
[cn4:17658] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22461218817
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 22461218817
[cn4:17658] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22461218817
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 22461218817
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 22461218817
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 22461218817
[cn4:17658] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22461218817
[cn4:17658] fetching data from INTERNAL dstore
[cn4:17658] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 22461218817
[cn4:17658] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22461218817
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 22461218817
[cn4:17658] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22461218817
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 31051153409
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 26756186113
[cn4:17658] dstore:hash:store: storing key[cn3:21245] [[15051,1],4]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21245] [[15051,1],4]:tcp:recv:handler CONNECTED
[cn3:21245] [[15051,1],4]:tcp:recv:handler allocate new recv msg
[cn3:21245] [[15051,1],4]:tcp:recv:handler read hdr
[cn3:21245] [[15051,1],4]:tcp:recv:handler allocate data region of size 14140
[cn1:14002] mca: base: components_register: registering dstore components
[cn1:14002] mca: base: components_register: found loaded component hash
[cn1:14002] mca: base: components_register: component hash register function successful
[cn1:14002] mca: base: components_open: opening dstore components
[cn1:14002] mca: base: components_open: found loaded component hash
[cn1:14002] mca:dstore:select: checking available component hash
[cn1:14002] mca:dstore:select: Querying component [hash]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 82
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 82 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 2348
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 2348 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 179
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 179 BYTES FOR DEST [[15051,0],0] TAG 5
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 82 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 2348 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 179 bytes from [[15051,0],2] for tag 5
[cn1:13986] [[15051,0],0] plm:base:receive processing msg
[cn1:13986] [[15051,0],0] routed_binomial: init routes for HNP job [15051,1]
[cn1:13986] [[15051,0],0] routed_binomial:callback got uri 986382337.4;tcp://192.168.1.3:49283 for job [15051,1] rank 4
[cn1:13986] [[15051,0],0] routed_binomial:callback got uri 986382337.5;tcp://192.168.1.3:58679 for job [15051,1] rank 5
[cn1:13986] [[15051,0],0] plm:base:receive done processing commands
[cn5:13995] [[15051,1],9]:tcp:recv:handler read hdr
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 22461218817
[cn2:00788] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22461218817
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 22461218817
[cn2:00788] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22461218817
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 22461218817
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 22461218817
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 22461218817
[cn2:00788] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22461218817
[cn2:00788] fetching data from INTERNAL dstore
[cn2:00788] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 22461218817
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22461218817
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 22461218817
[cn2:00788] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22461218817
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 26756186113
[cn2:00788] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26756186113
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 26756186113
[cn2:00788] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26756186113
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 26756186113
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 26756186113
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
EER dstore
[cn4:17657] dstore:hash:store storing data for proc 13871284225
[cn4:17657] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13871284225
[cn4:17657] fetching data from INTERNAL dstore
[cn4:17657] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 13871284225
[cn4:17657] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284225
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 13871284225
[cn4:17657] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13871284225
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 18166251521
[cn4:17657] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18166251521
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 18166251521
[cn4:17657] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18166251521
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 18166251521
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 18166251521
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 18166251521
[cn4:17657] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18166251521
[cn4:17657] fetching data from INTERNAL dstore
[cn4:17657] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 18166251521
[cn4:17657] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251521
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 18166251521
[cn4:17657] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18166251521
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 22461218817
[cn4:17657] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22461218817
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 22461218817
[cn4:17657] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22461218817
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 22461218817
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 22461218817
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 22461218817
[cn4:17657] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22461218817
[cn4:17657] fetching data from INTERNAL dstore
[cn4:17657] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 22461218817
[cn4:17657] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22461218817
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 22461218817
[cn4:17657] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22461218817
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 26756186113
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: storing key[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 108
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 108 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 2531
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 2531 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 273
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 273 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 108 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 2531 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 273 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 156
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 156 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 2352
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 2352 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 108
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 108 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] [[15051,1],9]:tcp:recv:handler allocate data region of size 14140
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 26756186113
[cn2:00788] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26756186113
[cn2:00788] fetching data from INTERNAL dstore
[cn2:00788] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 26756186113
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26756186113
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 26756186113
[cn2:00788] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26756186113
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 31051153409
[cn2:00788] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31051153409
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 31051153409
[cn2:00788] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31051153409
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 31051153409
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 31051153409
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 31051153409
[cn2:00788] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31051153409
[cn2:00788] fetching data from INTERNAL dstore
[cn2:00788] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 31051153409
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31051153409
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 31051153409
[cn2:00788] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31051153409
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 35346120705
[cn2:00788] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35346120705
[cn2:00788] storing data in INTERNAL dstore
 opal.local.rank[OPAL_UINT16] for proc 26756186113
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 26756186113
[cn4:17658] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26756186113
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 26756186113
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 26756186113
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 26756186113
[cn4:17658] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26756186113
[cn4:17658] fetching data from INTERNAL dstore
[cn4:17658] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 26756186113
[cn4:17658] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26756186113
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 26756186113
[cn4:17658] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26756186113
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31051153409
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31051153409
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31051153409
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 35346120705
[cn4:17658] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35346120705
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 35346120705
[cn4:17658] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35346120705
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 35346120705
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 35346120705
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 35346120705
[cn4:17658] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35346120705
[cn4:17658] fetching data from INTERNAL dstore
[cn4:17658] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 35346120705
[cn4:17658] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35346120705
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 35346120705
[cn4:17658] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35346120705
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 39641088001
[cn4:17658] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39641088001
[cn3:21246] [[15051,1],5]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21246] [[15051,1],5]:tcp:recv:handler CONNECTED
[cn3:21246] [[15051,1],5]:tcp:recv:handler allocate new recv msg
[cn3:21246] [[15051,1],5]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0] message received 156 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 2352 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 108 bytes from [[15051,0],2] for tag 2
[cn5:13994] [[15051,1],8]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13994] [[15051,1],8]:tcp:recv:handler CONNECTED
[cn2:00788] dstore:hash:store storing data for proc 35346120705
[cn2:00788] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35346120705
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 35346120705
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 35346120705
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 35346120705
[cn2:00788] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35346120705
[cn2:00788] fetching data from INTERNAL dstore
[cn2:00788] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 35346120705
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35346120705
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 35346120705
[cn2:00788] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35346120705
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 39641088001
[cn2:00788] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39641088001
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 39641088001
[cn2:00788] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39641088001
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 39641088001
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 39641088001
[cn2:00788] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 39641088001
[cn2:00788] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39641088001
[cn2:00788] fetching data from INTERNAL dstore
 opal.local.rank[OPAL_UINT16] for proc 26756186113
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26756186113
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26756186113
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 31051153409
[cn4:17657] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31051153409
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 31051153409
[cn4:17657] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31051153409
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 31051153409
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 31051153409
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 31051153409
[cn4:17657] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31051153409
[cn4:17657] fetching data from INTERNAL dstore
[cn4:17657] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 31051153409
[cn4:17657] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31051153409
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 31051153409
[cn4:17657] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31051153409
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 35346120705
[cn4:17657] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35346120705
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 35346120705
[cn4:17657] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35346120705
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 35346120705
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 35346120705
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 35346120705
[cn4:17657] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35346120705
[cn4:17657] fetching data from INTERNAL dstore
[cn4:17657] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 35346120705
[cn4:17657] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35346120705
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 35346120705
[cn4:17657] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35346120705
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 39641088001
[cn4:17657] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39641088001
[cn3:21246] [[15051,1],5]:tcp:recv:handler allocate data region of size 14140
[cn1:14002] mca: base: components_register: registering grpcomm components
[cn1:14002] mca: base: components_register: found loaded component bad
[cn1:14002] mca: base: components_register: component bad has no register or open function
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 429
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 429 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 563
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 563 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1880
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 1880 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 156
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 156 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 429 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 563 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 1880 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 156 bytes from [[15051,0],2] for tag 2
[cn5:13994] [[15051,1],8] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 14140 BYTES FOR DEST [[15051,1],8] TAG 20
[cn5:13994] [[15051,1],8] DELIVERING TO RML
[cn5:13994] [[15051,1],8] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] [[15051,1],8] message received 14140 bytes from [[15051,0],4] for tag 20
[cn2:00788] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 39641088001
[cn2:00788] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39641088001
[cn2:00788] storing data in INTERNAL dstore
[cn2:00788] dstore:hash:store storing data for proc 39641088001
[cn2:00788] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39641088001
[cn2:00788] [[15051,1],2] Process bound at launch
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 39641088001
[cn4:17657] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39641088001
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 39641088001
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 39641088001
[cn4:17657] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 39641088001
[cn4:17657] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39641088001
[cn4:17657] fetching data from INTERNAL dstore
[cn4:17657] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 39641088001
[cn4:17657] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39641088001
[cn4:17657] storing data in INTERNAL dstore
[cn4:17657] dstore:hash:store storing data for proc 39641088001
[cn4:17657] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39641088001
[cn4:17657] [[15051,1],6] Process bound at launch
[cn4:17657] MCW rank 6 bound to socket 0[core 0[hwt 0]]: [B/././.][./././.]
[cn4:17657] storing data in NONPEER dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn4:17657] storing data in PEER dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 26756186113
[cn3:21245] [[15051,1],4]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21245] [[15051,1],4]:tcp:recv:handler CONNECTED
[cn1:14002] mca: base: components_open: opening grpcomm components
[cn1:14002] mca: base: components_open: found loaded component bad
[cn1:14002] mca: base: components_open: component bad open function successful
[cn1:14002] mca:base:select: Auto-selecting grpcomm components
[cn1:14002] mca:base:select:(grpcomm) Querying component [bad]
[cn1:14002] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn1:14002] mca:base:select:(grpcomm) Selected component [bad]
[cn1:14002] [[15051,1],1] grpcomm:base:receive start comm
[cn1:14002] [[15051,1],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn1:14002] [[15051,1],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn1:14002] [[15051,1],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn1:14002] [[15051,1],1] setting up session dir with
	tmpdir: /tmp
	host cn1
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 83 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 301
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 301 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1880
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 1880 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 429
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 429 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 83 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 301 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 1880 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 429 bytes from [[15051,0],2] for tag 2
[cn5:13994] [[15051,1],8] registering sync ack recvd
[cn2:00788] MCW rank 2 bound to socket 0[core 0[hwt 0]]: [B/././.][./././.]
[cn2:00788] storing data in NONPEER dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 39641088001
[cn4:17658] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39641088001
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 39641088001
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 39641088001
[cn4:17658] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 39641088001
[cn4:17658] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39641088001
[cn4:17658] fetching data from INTERNAL dstore
[cn4:17658] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 39641088001
[cn4:17658] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39641088001
[cn4:17658] storing data in INTERNAL dstore
[cn4:17658] dstore:hash:store storing data for proc 39641088001
[cn4:17658] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39641088001
[cn4:17658] [[15051,1],7] Process bound at launch
[cn4:17658] MCW rank 7 bound to socket 0[core 1[hwt 0]]: [./B/./.][./././.]
[cn4:17658] storing data in NONPEER dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn4:17658] storing data in PEER dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 31051153409
[cn3:21245] [[15051,1],4] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 14140 BYTES FOR DEST [[15051,1],4] TAG 20
[cn3:21245] [[15051,1],4] DELIVERING TO RML
[cn3:21245] [[15051,1],4] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21245] [[15051,1],4] message received 14140 bytes from [[15051,0],2] for tag 20
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 103
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 103 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 231
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 231 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 464 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 83 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 103 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 231 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 464 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 83 bytes from [[15051,0],2] for tag 2
[cn5:13995] [[15051,1],9]:tcp:recv:handler called for peer [[15051,0],4]
[cn2:00788] storing data in PEER dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 9576316929
[cn4:17657] mca: base: components_register: registering bml components
[cn4:17657] mca: base: components_register: found loaded component r2
[cn4:17657] mca: base: components_register: component r2 register function successful
[cn4:17657] mca: base: components_open: opening bml components
[cn4:17657] mca: base: components_open: found loaded component r2
[cn4:17657] mca: base: components_open: component r2 open function successful
[cn3:21245] [[15051,1],4] registering sync ack recvd
[cn1:14002] [[15051,1],1] posting recv
[cn1:14002] [[15051,1],1] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn1:14002] [[15051,1],1] posting recv
[cn1:14002] [[15051,1],1] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn1:14002] [[15051,1],1] posting recv
[cn1:14002] [[15051,1],1] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 482
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 482 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 555
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 555 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 464 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 103
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 103 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 482 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 555 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 464 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 103 bytes from [[15051,0],2] for tag 2
[cn5:13995] [[15051,1],9]:tcp:recv:handler CONNECTED
[cn5:13995] [[15051,1],9] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 14140 BYTES FOR DEST [[15051,1],9] TAG 20
[cn5:13995] [[15051,1],9] DELIVERING TO RML
[cn5:13995] [[15051,1],9] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] [[15051,1],9] message received 14140 bytes from [[15051,0],4] for tag 20
[cn2:00789] [[15051,1],3]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00789] [[15051,1],3]:tcp:recv:handler CONNECTED
[cn2:00789] [[15051,1],3] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 14140 BYTES FOR DEST [[15051,1],3] TAG 20
[cn2:00789] [[15051,1],3] DELIVERING TO RML
[cn2:00789] [[15051,1],3] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00789] [[15051,1],3] message received 14140 bytes from [[15051,0],1] for tag 20
[cn4:17658] mca: base: components_register: registering bml components
[cn4:17658] mca: base: components_register: found loaded component r2
[cn4:17658] mca: base: components_register: component r2 register function successful
[cn4:17658] mca: base: components_open: opening bml components
[cn4:17658] mca: base: components_open: found loaded component r2
[cn4:17658] mca: base: components_open: component r2 open function successful
[cn3:21246] [[15051,1],5]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:14002] procdir: /tmp/openmpi-sessions-artpol@cn1_0/15051/1/1
[cn1:14002] jobdir: /tmp/openmpi-sessions-artpol@cn1_0/15051/1
[cn1:14002] top: openmpi-sessions-artpol@cn1_0
[cn1:14002] tmp: /tmp
[cn1:14002] storing data in INTERNAL dstore
[cn1:14002] dstore:hash:store storing data for proc 5281349633
[cn1:14002] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 5281349633
[cn1:14002] storing data in INTERNAL dstore
[cn1:14002] dstore:hash:store storing data for proc 5281349633
[cn1:14002] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 5281349633
[cn1:14002] [[15051,1],1] routed_binomial: init routes for proc job [15051,1]
	hnp_uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
	daemon uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn1:14002] [[15051,1],1] [[15051,0],0] with daemon registering
[cn1:14002] [[15051,1],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn1:14002] [[15051,1],1] rml_send_buffer to peer [[15051,0],0] at tag 1
[cn1:14002] [[15051,1],1] registering sync waiting for ack
[cn1:14002] [[15051,1],1] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn1:14002] [[15051,1],1]: set_addr to uri 986382336.0;tcp://192.168.1.1,192.168.10.2:37478
[cn1:14002] [[15051,1],1]:set_addr checking if peer [[15051,0],0] is reachable via component tcp
[cn1:14002] [[15051,1],1] oob:tcp: working peer [[15051,0],0] address tcp://192.168.1.1,192.168.10.2:37478
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 83 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 83 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 2016
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 2016 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 943
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 943 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 83 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 83 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 2016 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 943 bytes from [[15051,0],2] for tag 2
[cn5:13995] [[15051,1],9] registering sync ack recvd
[cn2:00789] [[15051,1],3] registering sync ack recvd
[cn4:17657] mca: base: components_register: registering btl components
[cn4:17657] mca: base: components_register: found loaded component openib
[cn4:17657] mca: base: components_register: component openib register function successful
[cn4:17657] mca: base: components_register: found loaded component self
[cn4:17657] mca: base: components_register: component self register function successful
[cn4:17657] mca: base: components_register: found loaded component sm
[cn4:17657] mca: base: components_register: component sm register function successful
[cn4:17657] mca: base: components_register: found loaded component tcp
[cn4:17657] mca: base: components_register: component tcp register function successful
[cn4:17657] mca: base: components_register: found loaded component usnic
[cn4:17657] mca: base: components_register: component usnic register function successful
[cn4:17657] mca: base: components_register: found loaded component vader
[cn4:17657] mca: base: components_register: component vader register function successful
[cn4:17657] mca: base: components_open: opening btl components
[cn4:17657] mca: base: components_open: found loaded component openib
[cn4:17657] mca: base: components_open: component openib open function successful
[cn4:17657] mca: base: components_open: found loaded component self
[cn4:17657] mca: base: components_open: component self open function successful
[cn4:17657] mca: base: components_open: found loaded component sm
[cn4:17657] mca: base: components_open: component sm open function successful
[cn4:17657] mca: base: components_open: found loaded component tcp
[cn4:17657] mca: base: components_open: component tcp open function successful
[cn4:17657] mca: base: components_open: found loaded component usnic
[cn4:17657] mca: base: components_open: component usnic open function successful
[cn4:17657] mca: base: components_open: found loaded component vader
[cn4:17657] mca: base: components_open: component vader open function successful
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 986382336
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 986382336
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21245] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 5281349632
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349632
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21245] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 9576316928
[cn1:14002] [[15051,1],1] PEER [[15051,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:14002] [[15051,1],1] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:14002] [[15051,1],1]:tcp set addr for peer [[15051,0],0]
[cn1:14002] [[15051,1],1] PEER [[15051,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 3 INTERFACE eth1
[cn1:14002] [[15051,1],1] PASSING ADDR 192.168.10.2 TO INTERFACE eth1 AT KERNEL INDEX 3
[cn1:14002] [[15051,1],1]:tcp set addr for peer [[15051,0],0]
[cn1:14002] [[15051,1],1]: peer [[15051,0],0] is reachable via component tcp
[cn1:14002] [[15051,1],1] posting recv
[cn1:14002] [[15051,1],1] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn1:14002] [[15051,1],1] rml_send_msg to peer [[15051,0],0] at tag 1
[cn1:14002] [[15051,1],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14002] [[15051,1],1]:tcp:processing set_peer cmd for interface eth0
[cn1:14002] [[15051,1],1] SET_PEER ADDING PEER [[15051,0],0]
[cn1:14002] [[15051,1],1] set_peer: peer [[15051,0],0] is listening on interface eth0 net 192.168.1.1 port 37478
[cn1:14002] [[15051,1],1]:tcp:processing set_peer cmd for interface eth1
[cn1:14002] [[15051,1],1] SET_PEER ADDING PEER [[15051,0],0]
[cn1:14002] [[15051,1],1] set_peer: peer [[15051,0],0] is listening on interface eth1 net 192.168.10.2 port 37478
[cn1:14002] [[15051,1],1] oob:base:send to target [[15051,0],0]
[cn1:14002] [[15051,1],1] oob:tcp:send_nb to peer [[15051,0],0]:1
[cn1:14002] [[15051,1],1] tcp:send_nb to peer [[15051,0],0]
[cn1:14002] [[15051,1],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:14002] [[15051,1],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:1
[cn1:14002] [[15051,1],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:14002] [[15051,1],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15051,0],0]
[cn1:14002] [[15051,1],1] tcp:send_nb: initiating connection to [[15051,0],0]
[cn1:14002] [[15051,1],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0] mca_oob_tcp_listen_thread: new connection: (23, 11)[cn1:13986] [[15051,0],0] mca_oob_tcp_listen_thread: new connection: (23, 11) 192.168.1.1:59870
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 2016
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 2016 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1969
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1969 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] connection_handler: working connection (23, 11) 192.168.1.1:59870
[cn1:13986] [[15051,0],0] accept_connection: 192.168.1.1:59870
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 2016 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 1969 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called
[cn1:13986] [[15051,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 23
[cn1:13986] [[15051,0],0] waiting for connect ack from UNKNOWN
[cn1:13986] [[15051,0],0] connect ack received from UNKNOWN
[cn1:13986] [[15051,0],0] connect-ack recvd from UNKNOWN
[cn1:13986] [[15051,0],0] mca_oob_tcp_recv_connect: connection from new peer
[cn1:13986] [[15051,0],0] connect-ack header from [[15051,1],1] is okay
[cn1:13986] [[15051,0],0] waiting for connect ack from [[15051,1],1]
[cn1:13986] [[15051,0],0] connect ack received from [[15051,1],1]
[cn1:13986] [[15051,0],0] connect-ack version from [[15051,1],1] matches ours
[cn1:13986] [[15051,0],0] connect-ack [[15051,1],1] authenticated
[cn1:13986] [[15051,0],0] tcp:peer_accept called for peer [[15051,1],1] in state UNKNOWN on socket 23
[cn1:13986] [[15051,0],0] SEND CONNECT ACK
[cn1:13986] [[15051,0],0] send blocking of 47 bytes to socket 23
[cn1:13986] [[15051,0],0] blocking send complete to socket 23
[cn1:13986] [[15051,0],0]-[[15051,1],1] tcp_peer_connected on socket 23
[cn1:13986] [[15051,0],0] routed_binomial_update: [[15051,1],1] --> [[15051,1],1]
[cn1:13986] [[15051,0],0]-[[15051,1],1] accepted: 192.168.1.1 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 3034
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 3034 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 986382336
[cn5:13995] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 986382336
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 18446744069414584318
[cn5:13995] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 5281349632
[cn5:13995] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349632
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 18446744069414584318
[cn5:13995] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 9576316928
[cn5:13995] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316928
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 18446744069414584318
[cn5:13995] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 13871284224
[cn5:13995] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284224
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 18446744069414584318
[cn5:13995] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 18166251520
[cn5:13995] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251520
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 18446744069414584318
[cn5:13995] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 39641088001
[cn5:13995] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39641088001
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 39641088001
[cn5:13995] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39641088001
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 18446744070400966657
[cn5:13995] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 18446744070400966657
[cn5:13995] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 986382337
[cn5:13995] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 986382337
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 986382337
[cn5:13995] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 986382337
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 986382337
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 986382337
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 986382337
[cn5:13995] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 986382337
[cn5:13995] fetching data from INTERNAL dstore
[cn5:13995] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 986382337
[cn5:13995] dstore:hash:store: stori[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 986382336
[cn2:00789] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 986382336
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00789] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 5281349632
[cn2:00789] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349632
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00789] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284225
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13871284225
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 9576316928
[cn2:00789] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316928
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00789] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284224
[cn2:00789] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284224
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00789] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 18166251520
[cn2:00789] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251520
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00789] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 18446744070400966657
[cn2:00789] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 18446744070400966657
[cn2:00789] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 986382337
[cn2:00789] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 986382337
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 986382337
[cn2:00789] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 986382337
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 986382337
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 986382337
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 986382337
[cn2:00789] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 986382337
[cn2:00789] fetching data from INTERNAL dstore
[cn2:00789] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 986382337
[cn2:00789] dstore:hash:store: stori[cn4:17658] mca: base: components_register: registering btl components
[cn4:17658] mca: base: components_register: found loaded component openib
[cn4:17658] mca: base: components_register: component openib register function successful
[cn4:17658] mca: base: components_register: found loaded component self
[cn4:17658] mca: base: components_register: component self register function successful
[cn4:17658] mca: base: components_register: found loaded component sm
[cn4:17658] mca: base: components_register: component sm register function successful
[cn4:17658] mca: base: components_register: found loaded component tcp
[cn4:17658] mca: base: components_register: component tcp register function successful
[cn4:17658] mca: base: components_register: found loaded component usnic
[cn4:17658] mca: base: components_register: component usnic register function successful
[cn4:17658] mca: base: components_register: found loaded component vader
[cn4:17658] mca: base: components_register: component vader register function successful
[cn4:17658] mca: base: components_open: opening btl components
[cn4:17658] mca: base: components_open: found loaded component openib
[cn4:17658] mca: base: components_open: component openib open function successful
[cn4:17658] mca: base: components_open: found loaded component self
[cn4:17658] mca: base: components_open: component self open function successful
[cn4:17658] mca: base: components_open: found loaded component sm
[cn4:17658] mca: base: components_open: component sm open function successful
[cn4:17658] mca: base: components_open: found loaded component tcp
[cn4:17658] mca: base: components_open: component tcp open function successful
[cn4:17658] mca: base: components_open: found loaded component usnic
[cn4:17658] mca: base: components_open: component usnic open function successful
[cn4:17658] mca: base: components_open: found loaded component vader
[cn4:17658] mca: base: components_open: component vader open function successful
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316928
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21245] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251521
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18166251521
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 13871284224
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284224
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21245] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251520
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251520
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21245] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 18446744070400966657
[cn3:21245] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 18446744070400966657
[cn3:21245] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn1:14002] [[15051,1],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] via interface eth0
[cn1:14002] [[15051,1],1] oob:tcp:peer creating socket to [[15051,0],0]
[cn1:14002] [[15051,1],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] via interface eth0 on socket 11
[cn1:14002] [[15051,1],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15051,0],0] on 192.168.1.1:37478 - 0 retries
[cn1:14002] [[15051,1],1] waiting for connect completion to [[15051,0],0] - activating send event
[cn1:14002] [[15051,1],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:14002] [[15051,1],1] tcp:send_handler CONNECTING
[cn1:14002] [[15051,1],1]:tcp:complete_connect called for peer [[15051,0],0] on socket 11
[cn1:14002] [[15051,1],1] tcp_peer_complete_connect: sending ack to [[15051,0],0]
[cn1:14002] [[15051,1],1] SEND CONNECT ACK
[cn1:14002] fetching data from INTERNAL dstore
[cn1:14002] dstore:hash:fetch: searching for key opal.cred on proc 5281349633
[cn1:14002] dstore_hash:fetch key opal.cred for proc 5281349633 not found
[cn1:14002] [[15051,1],1] send blocking of 47 bytes to socket 11
[cn1:14002] [[15051,1],1] blocking send complete to socket 11
[cn1:14002] [[15051,1],1] tcp_peer_complete_connect: setting read event on connection to [[15051,0],0]
[cn1:14002] [[15051,1],1]:tcp:recv:handler called for peer [[15051,0],0]
[cn1:14002] [[15051,1],1] RECV CONNECT ACK FROM [[15051,0],0] ON SOCKET 11
[cn1:14002] [[15051,1],1] waiting for connect ack from [[15051,0],0]
[cn1:14002] [[15051,1],1] connect ack received from [[15051,0],0]
[cn1:14002] [[15051,1],1] connect-ack recvd from [[15051,0],0]
[cn1:14002] [[15051,1],1] connect-ack header from [[15051,0],0] is okay
[cn1:14002] [[15051,1],1] waiting for connect ack from [[15051,0],0]
[cn1:14002] [[15051,1],1] connect ack received from [[15051,0],0]
[cn1:14002] [[15051,1],1] connect-ack version from [[15051,0],0] matches ours
[cn1:14002] [[15051,1],1] connect-ack [[15051,0],0] authenticated
[cn1:14002] [[15051,1],1]-[[15051,0],0] tcp_peer_connected on socket 11
[cn1:14002] [[15051,1],1]-[[15051,0],0] connected: 192.168.1.1 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:14002] [[15051,1],1]:tcp:recv:handler starting send/recv events
[cn1:14002] [[15051,1],1] tcp:set_module called for peer [[15051,0],0]
[cn1:14002] [[15051,1],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:14002] [[15051,1],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:14002] [[15051,1],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 74 BYTES ON SOCKET 11
[cn1:14002] [[15051,1],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] tcp:set_module called for peer [[15051,1],1]
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 3034 bytes from [[15051,0],2] for tag 2
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 986382336
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 986382336
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 18446744069414584318
[cn5:13994] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 5281349632
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349632
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 18446744069414584318
[cn5:13994] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 9576316928
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316928
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 18446744069414584318
[cn5:13994] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 13871284224
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284224
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 18446744069414584318
[cn5:13994] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 18166251520
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251520
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 18446744069414584318
[cn5:13994] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35346120705
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35346120705
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 18446744070400966657
[cn5:13994] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 18446744070400966657
[cn5:13994] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 986382337
[cn5:13994] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 986382337
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 986382337
[cn5:13994] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 986382337
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 986382337
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 986382337
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 986382337
[cn5:13994] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 986382337
[cn5:13994] fetching data from INTERNAL dstore
[cn5:13994] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 986382337
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 986382337
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 986382337
[cn2:00789] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 986382337
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 5281349633
[cn2:00789] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5281349633
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 5281349633
[cn2:00789] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5281349633
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 5281349633
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 5281349633
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 5281349633
[cn2:00789] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5281349633
[cn2:00789] fetching data from INTERNAL dstore
[cn2:00789] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 5281349633
[cn2:00789] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349633
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 5281349633
[cn2:00789] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5281349633
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 13871284225
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 9576316929
[cn2:00789] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9576316929
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 9576316929
[cn2:00789] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9576316929
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 9576316929
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 9576316929
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 9576316929
[cn2:00789] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9576316929
[cn2:00789] fetching data from INTERNAL dstore
[cn2:00789] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 9576316929
[cn2:00789] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316929
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 9576316929
[cn2:00789] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9576316929
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13871284225
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13871284225
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn2:00789] storing data in [cn4:17657] mca: base: components_register: registering pml components
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 986382337
[cn3:21245] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 986382337
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 986382337
[cn3:21245] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 986382337
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 986382337
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 986382337
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 986382337
[cn3:21245] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 986382337
[cn3:21245] fetching data from INTERNAL dstore
[cn3:21245] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 986382337
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 986382337
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 986382337
[cn3:21245] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 986382337
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 5281349633
[cn3:21245] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5281349633
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 5281349633
[cn3:21245] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5281349633
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 5281349633
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 5281349633
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 5281349633
[cn3:21245] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5281349633
[cn3:21245] fetching data from INTERNAL dstore
[cn3:21245] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 5281349633
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349633
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 5281349633
[cn3:21245] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5281349633
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 326
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 326 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1849
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1849 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,1],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 74
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 326 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 1849 bytes from [[15051,0],2] for tag 2
ng key orte.hostname[OPAL_STRING] for proc 986382337
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 986382337
[cn5:13995] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 986382337
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 5281349633
[cn5:13995] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5281349633
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 5281349633
[cn5:13995] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5281349633
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 5281349633
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 5281349633
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 5281349633
[cn5:13995] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5281349633
[cn5:13995] fetching data from INTERNAL dstore
[cn5:13995] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 5281349633
[cn5:13995] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349633
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 5281349633
[cn5:13995] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5281349633
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 9576316929
[cn5:13995] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9576316929
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 9576316929
[cn5:13995] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9576316929
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 9576316929
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 9576316929
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 9576316929
[cn5:13995] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9576316929
[cn5:13995] fetching data from INTERNAL dstore
[cn5:13995] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 9576316929
[cn5:13995] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316929
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 9576316929
[cn5:13995] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9576316929
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 13871284225
[cn5:13995] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13871284225
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 13871284225
[cn5:13995] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13871284225
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 13871284225
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 13871284225
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn5:13995] storing data in NONPNONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13871284225
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 18166251521
[cn2:00789] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18166251521
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 18166251521
[cn2:00789] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18166251521
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 18166251521
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 18166251521
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 18166251521
[cn2:00789] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18166251521
[cn2:00789] fetching data from INTERNAL dstore
[cn2:00789] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 18166251521
[cn2:00789] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251521
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 18166251521
[cn2:00789] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18166251521
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 22461218817
[cn2:00789] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22461218817
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 22461218817
[cn2:00789] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22461218817
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 22461218817
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 22461218817
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 22461218817
[cn2:00789] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22461218817
[cn2:00789] fetching data from INTERNAL dstore
[cn2:00789] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 22461218817
[cn2:00789] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22461218817
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 22461218817
[cn2:00789] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22461218817
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 26756186113
[cn2:00789] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26756186113
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 26756186113
[cn2:00789] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26756186113
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 26756186113
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing [cn4:17658] mca: base: components_register: registering pml components
[cn4:17658] mca: base: components_register: found loaded component v
[cn4:17658] mca: base: components_register: component v register function successful
[cn4:17658] mca: base: components_register: found loaded component bfo
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 9576316929
[cn3:21245] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9576316929
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 9576316929
[cn3:21245] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9576316929
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 9576316929
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 9576316929
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 9576316929
[cn3:21245] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9576316929
[cn3:21245] fetching data from INTERNAL dstore
[cn3:21245] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 9576316929
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316929
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 9576316929
[cn3:21245] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9576316929
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 13871284225
[cn3:21245] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13871284225
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 13871284225
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 3081
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 3081 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 3081 bytes from [[15051,0],2] for tag 2
ng key orte.hostname[OPAL_STRING] for proc 986382337
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 986382337
[cn5:13994] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 986382337
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 5281349633
[cn5:13994] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5281349633
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 5281349633
[cn5:13994] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5281349633
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 5281349633
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 5281349633
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 5281349633
[cn5:13994] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5281349633
[cn5:13994] fetching data from INTERNAL dstore
[cn5:13994] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 5281349633
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349633
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 5281349633
[cn5:13994] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5281349633
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 9576316929
[cn5:13994] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9576316929
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 9576316929
[cn5:13994] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9576316929
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 9576316929
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 9576316929
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 9576316929
[cn5:13994] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9576316929
[cn5:13994] fetching data from INTERNAL dstore
[cn5:13994] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 9576316929
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316929
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 9576316929
[cn5:13994] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9576316929
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 13871284225
[cn5:13994] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13871284225
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 13871284225
[cn5:13994] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13871284225
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 13871284225
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 13871284225
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn5:13994] storing data in NONPdata for proc 26756186113
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 26756186113
[cn2:00789] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26756186113
[cn2:00789] fetching data from INTERNAL dstore
[cn2:00789] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 26756186113
[cn2:00789] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26756186113
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 26756186113
[cn2:00789] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26756186113
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 31051153409
[cn2:00789] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31051153409
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 31051153409
[cn2:00789] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31051153409
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 31051153409
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 31051153409
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 31051153409
[cn2:00789] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31051153409
[cn2:00789] fetching data from INTERNAL dstore
[cn2:00789] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 31051153409
[cn2:00789] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31051153409
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 31051153409
[cn2:00789] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31051153409
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 35346120705
[cn2:00789] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35346120705
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 35346120705
[cn2:00789] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35346120705
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 35346120705
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 35346120705
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 35346120705
[cn2:00789] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35346120705
[cn2:00789] fetching data from INTERNAL dstore
[cn2:00789] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 35346120705
[cn2:00789] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35346120705
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 35346120705
[cn2:00789] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35346120705
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 39641088001
[cn2:00789] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39641088001[cn4:17658] mca: base: components_register: component bfo register function successful
[cn3:21245] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13871284225
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 13871284225
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 13871284225
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 13871284225
[cn3:21245] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13871284225
[cn3:21245] fetching data from INTERNAL dstore
[cn3:21245] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 13871284225
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284225
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 13871284225
[cn3:21245] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13871284225
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 18166251521
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18166251521
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18166251521
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18166251521
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 22461218817
[cn3:21245] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22461218817
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 22461218817
[cn3:21245] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22461218817
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 22461218817
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn3:21245] storing data in NONPEER dstore
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1881
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 1881 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 186
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 186 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1814
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1814 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 1881 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 186 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 1814 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 464 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 3304
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 3304 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
EER dstore
[cn5:13995] dstore:hash:store storing data for proc 13871284225
[cn5:13995] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13871284225
[cn5:13995] fetching data from INTERNAL dstore
[cn5:13995] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 13871284225
[cn5:13995] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284225
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 13871284225
[cn5:13995] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13871284225
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 18166251521
[cn5:13995] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18166251521
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 18166251521
[cn5:13995] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18166251521
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 18166251521
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 18166251521
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 18166251521
[cn5:13995] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18166251521
[cn5:13995] fetching data from INTERNAL dstore
[cn5:13995] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 18166251521
[cn5:13995] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251521
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 18166251521
[cn5:13995] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18166251521
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 22461218817
[cn5:13995] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22461218817
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 22461218817
[cn5:13995] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22461218817
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 22461218817
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 22461218817
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 22461218817
[cn5:13995] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22461218817
[cn5:13995] fetching data from INTERNAL dstore
[cn5:13995] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 22461218817
[cn5:13995] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22461218817
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 22461218817
[cn5:13995] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22461218817
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 26756186113
[cn5:13995] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26756186113
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 26756186113
[cn5:13995] dstore:hash:store: storing ke
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 39641088001
[cn2:00789] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39641088001
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 39641088001
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 39641088001
[cn2:00789] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 39641088001
[cn2:00789] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39641088001
[cn2:00789] fetching data from INTERNAL dstore
[cn2:00789] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 39641088001
[cn2:00789] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39641088001
[cn2:00789] storing data in INTERNAL dstore
[cn2:00789] dstore:hash:store storing data for proc 39641088001
[cn2:00789] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39641088001
[cn2:00789] [[15051,1],3] Process bound at launch
[cn2:00789] MCW rank 3 bound to socket 0[core 1[hwt 0]]: [./B/./.][./././.]
[cn2:00789] storing data in NONPEER dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn2:00789] storing data in PEER dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 13871284225
[cn4:17658] mca: base: components_register: found loaded component cm
[cn4:17658] mca: base: components_register: component cm register function successful
[cn3:21245] dstore:hash:store storing data for proc 22461218817
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 22461218817
[cn3:21245] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22461218817
[cn3:21245] fetching data from INTERNAL dstore
[cn3:21245] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 22461218817
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22461218817
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 22461218817
[cn3:21245] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22461218817
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 26756186113
[cn3:21245] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26756186113
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 26756186113
[cn3:21245] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26756186113
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 26756186113
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 26756186113
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 26756186113
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 464 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 3304 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 464 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 245
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 245 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1555
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1555 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
EER dstore
[cn5:13994] dstore:hash:store storing data for proc 13871284225
[cn5:13994] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13871284225
[cn5:13994] fetching data from INTERNAL dstore
[cn5:13994] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 13871284225
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284225
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 13871284225
[cn5:13994] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13871284225
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 18166251521
[cn5:13994] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18166251521
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 18166251521
[cn5:13994] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18166251521
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 18166251521
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 18166251521
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 18166251521
[cn5:13994] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18166251521
[cn5:13994] fetching data from INTERNAL dstore
[cn5:13994] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 18166251521
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251521
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 18166251521
[cn5:13994] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18166251521
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 22461218817
[cn5:13994] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22461218817
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 22461218817
[cn5:13994] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22461218817
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 22461218817
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 22461218817
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 22461218817
[cn5:13994] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22461218817
[cn5:13994] fetching data from INTERNAL dstore
[cn5:13994] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 22461218817
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22461218817
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 22461218817
[cn5:13994] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22461218817
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 26756186113
[cn5:13994] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26756186113
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 26756186113
[cn5:13994] dstore:hash:store: storing ke[cn2:00788] mca: base: components_register: registering bml components
[cn2:00788] mca: base: components_register: found loaded component r2
[cn2:00788] mca: base: components_register: component r2 register function successful
[cn2:00788] mca: base: components_open: opening bml components
[cn2:00788] mca: base: components_open: found loaded component r2
[cn2:00788] mca: base: components_open: component r2 open function successful
[cn4:17658] mca: base: components_register: found loaded component ob1
[cn3:21245] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26756186113
[cn3:21245] fetching data from INTERNAL dstore
[cn3:21245] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 26756186113
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26756186113
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 26756186113
[cn3:21245] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26756186113
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 31051153409
[cn3:21245] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31051153409
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 31051153409
[cn3:21245] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31051153409
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 31051153409
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 31051153409
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 31051153409
[cn3:21245] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31051153409
[cn3:21245] fetching data from INTERNAL dstore
[cn3:21245] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 31051153409
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31051153409
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 31051153409
[cn3:21245] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31051153409
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 35346120705
[cn3:21245] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35346120705
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 35346120705
[cn3:21245] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35346120705
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 35346120705
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 35346120705
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 35346120705
[cn3:21245] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35346120705
[cn3:21245] fetching data from INTERNAL dstore
[cn3:21245] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 464 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 245 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 1555 bytes from [[15051,0],2] for tag 2
y orte.node.rank[OPAL_UINT16] for proc 26756186113
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 26756186113
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 26756186113
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 26756186113
[cn5:13995] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26756186113
[cn5:13995] fetching data from INTERNAL dstore
[cn5:13995] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 26756186113
[cn5:13995] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26756186113
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 26756186113
[cn5:13995] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26756186113
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 31051153409
[cn5:13995] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31051153409
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 31051153409
[cn5:13995] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31051153409
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 31051153409
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 31051153409
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 31051153409
[cn5:13995] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31051153409
[cn5:13995] fetching data from INTERNAL dstore
[cn5:13995] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 31051153409
[cn5:13995] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31051153409
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 31051153409
[cn5:13995] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31051153409
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 39641088001
[cn5:13995] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 39641088001
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 35346120705
[cn5:13995] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35346120705
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 35346120705
[cn5:13995] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35346120705
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 35346120705
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 35346120705
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 35346120705
[cn5:13995] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35346120705
[cn5:13995] fetching data from INTERNAL dstore
[cn5:13995] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:stor[cn2:00789] mca: base: components_register: registering bml components
[cn2:00789] mca: base: components_register: found loaded component r2
[cn2:00789] mca: base: components_register: component r2 register function successful
[cn2:00789] mca: base: components_open: opening bml components
[cn2:00789] mca: base: components_open: found loaded component r2
[cn2:00789] mca: base: components_open: component r2 open function successful
[cn4:17658] mca: base: components_register: component ob1 register function successful
[cn4:17658] mca: base: components_open: opening pml components
[cn4:17658] mca: base: components_open: found loaded component v
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 35346120705
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35346120705
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 35346120705
[cn3:21245] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35346120705
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 39641088001
[cn3:21245] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39641088001
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 39641088001
[cn3:21245] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39641088001
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 39641088001
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 39641088001
[cn3:21245] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 39641088001
[cn3:21245] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39641088001
[cn3:21245] fetching data from INTERNAL dstore
[cn3:21245] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 2016
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 2016 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 184
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 184 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 480
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 480 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 2016 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 184 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 480 bytes from [[15051,0],2] for tag 2
y orte.node.rank[OPAL_UINT16] for proc 26756186113
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 26756186113
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 26756186113
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 26756186113
[cn5:13994] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26756186113
[cn5:13994] fetching data from INTERNAL dstore
[cn5:13994] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 26756186113
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26756186113
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 26756186113
[cn5:13994] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26756186113
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 31051153409
[cn5:13994] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31051153409
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 31051153409
[cn5:13994] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31051153409
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 31051153409
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 31051153409
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 31051153409
[cn5:13994] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31051153409
[cn5:13994] fetching data from INTERNAL dstore
[cn5:13994] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 31051153409
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31051153409
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 31051153409
[cn5:13994] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31051153409
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 35346120705
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35346120705
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35346120705
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35346120705
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 39641088001
[cn5:13994] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39641088001
[cn2:00788] mca: base: components_register: registering btl components
[cn2:00788] mca: base: components_register: found loaded component openib
[cn2:00788] mca: base: components_register: component openib register function successful
[cn2:00788] mca: base: components_register: found loaded component self
[cn2:00788] mca: base: components_register: component self register function successful
[cn2:00788] mca: base: components_register: found loaded component sm
[cn2:00788] mca: base: components_register: component sm register function successful
[cn2:00788] mca: base: components_register: found loaded component tcp
[cn2:00788] mca: base: components_register: component tcp register function successful
[cn2:00788] mca: base: components_register: found loaded component usnic
[cn2:00788] mca: base: components_register: component usnic register function successful
[cn2:00788] mca: base: components_register: found loaded component vader
[cn2:00788] mca: base: components_register: component vader register function successful
[cn2:00788] mca: base: components_open: opening btl components
[cn2:00788] mca: base: components_open: found loaded component openib
[cn2:00788] mca: base: components_open: component openib open function successful
[cn2:00788] mca: base: components_open: found loaded component self
[cn2:00788] mca: base: components_open: component self open function successful
[cn2:00788] mca: base: components_open: found loaded component sm
[cn2:00788] mca: base: components_open: component sm open function successful
[cn2:00788] mca: base: components_open: found loaded component tcp
[cn2:00788] mca: base: components_open: component tcp open function successful
[cn2:00788] mca: base: components_open: found loaded component usnic
[cn2:00788] mca: base: components_open: component usnic open function successful
[cn2:00788] mca: base: components_open: found loaded component vader
[cn2:00788] mca: base: components_open: component vader open function successful
[cn4:17657] mca: base: components_register: found loaded component v
[cn4:17657] mca: base: components_register: component v register function successful
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 39641088001
[cn3:21245] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39641088001
[cn3:21245] storing data in INTERNAL dstore
[cn3:21245] dstore:hash:store storing data for proc 39641088001
[cn3:21245] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39641088001
[cn3:21245] [[15051,1],4] Process bound at launch
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1880
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1880 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 2016
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 2016 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 343
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 343 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 1880 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 2016 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 343 bytes from [[15051,0],2] for tag 2
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 39641088001
[cn5:13994] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39641088001
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 39641088001
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 39641088001
[cn5:13994] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 39641088001
[cn5:13994] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39641088001
[cn5:13994] fetching data from INTERNAL dstore
[cn5:13994] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 39641088001
[cn5:13994] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39641088001
[cn5:13994] storing data in INTERNAL dstore
[cn5:13994] dstore:hash:store storing data for proc 39641088001
[cn5:13994] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39641088001
[cn5:13994] [[15051,1],8] Process bound at launch
[cn5:13994] MCW rank 8 bound to socket 0[core 0[hwt 0]]: [B/././.][./././.]
[cn5:13994] storing data in NONPEER dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn5:13994] storing data in PEER dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 35346120705
[cn2:00789] mca: base: components_register: registering btl components
[cn2:00789] mca: base: components_register: found loaded component openib
[cn2:00789] mca: base: components_register: component openib register function successful
[cn2:00789] mca: base: components_register: found loaded component self
[cn2:00789] mca: base: components_register: component self register function successful
[cn2:00789] mca: base: components_register: found loaded component sm
[cn2:00789] mca: base: components_register: component sm register function successful
[cn2:00789] mca: base: components_register: found loaded component tcp
[cn2:00789] mca: base: components_register: component tcp register function successful
[cn2:00789] mca: base: components_register: found loaded component usnic
[cn2:00789] mca: base: components_register: component usnic register function successful
[cn2:00789] mca: base: components_register: found loaded component vader
[cn2:00789] mca: base: components_register: component vader register function successful
[cn2:00789] mca: base: components_open: opening btl components
[cn2:00789] mca: base: components_open: found loaded component openib
[cn2:00789] mca: base: components_open: component openib open function successful
[cn2:00789] mca: base: components_open: found loaded component self
[cn2:00789] mca: base: components_open: component self open function successful
[cn2:00789] mca: base: components_open: found loaded component sm
[cn2:00789] mca: base: components_open: component sm open function successful
[cn2:00789] mca: base: components_open: found loaded component tcp
[cn2:00789] mca: base: components_open: component tcp open function successful
[cn2:00789] mca: base: components_open: found loaded component usnic
[cn2:00789] mca: base: components_open: component usnic open function successful
[cn2:00789] mca: base: components_open: found loaded component vader
[cn2:00789] mca: base: components_open: component vader open function successful
[cn4:17657] mca: base: components_register: found loaded component bfo
[cn3:21245] MCW rank 4 bound to socket 0[core 0[hwt 0]]: [B/././.][./././.]
[cn3:21245] storing data in NONPEER dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn3:21245] storing data in PEER dstore
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1880
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1880 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 855
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 855 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 273
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 273 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 193
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 193 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 1880 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 855 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 273 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 193 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 464 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 855
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 855 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 482
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 482 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
e storing data for proc 35346120705
[cn5:13995] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35346120705
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 35346120705
[cn5:13995] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35346120705
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 39641088001
[cn5:13995] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39641088001
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 39641088001
[cn5:13995] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39641088001
[cn5:13995] storing data in INTERNAL dstore
[cn5:13995] dstore:hash:store storing data for proc 39641088001
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 39641088001
[cn5:13995] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 39641088001
[cn5:13995] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39641088001
[cn5:13995] [[15051,1],9] Process bound at launch
[cn5:13995] MCW rank 9 bound to socket 0[core 1[hwt 0]]: [./B/./.][./././.]
[cn5:13995] storing data in NONPEER dstore
[cn5:13995] dstore:hash:store storing data for proc 39641088001
[cn5:13995] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn5:13995] storing data in PEER dstore
[cn5:13995] dstore:hash:store storing data for proc 39641088001
[cn5:13995] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 39641088001
[cn2:00788] mca: base: components_register: registering pml components
[cn2:00788] mca: base: components_register: found loaded component v
[cn2:00788] mca: base: components_register: component v register function successful
[cn2:00788] mca: base: components_register: found loaded component bfo
[cn2:00788] mca: base: components_register: component bfo register function successful
[cn2:00788] mca: base: components_register: found loaded component cm
[cn2:00788] mca: base: components_register: component cm register function successful
[cn2:00788] mca: base: components_register: found loaded component ob1
[cn2:00788] mca: base: components_register: component ob1 register function successful
[cn2:00788] mca: base: components_open: opening pml components
[cn2:00788] mca: base: components_open: found loaded component v
[cn4:17657] mca: base: components_register: component bfo register function successful
[cn4:17657] mca: base: components_register: found loaded component cm
[cn4:17657] mca: base: components_register: component cm register function successful
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 18166251521
[cn1:13986] [[15051,0],0] message received 464 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 855 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 482 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: registering bml components
[cn5:13994] mca: base: components_register: found loaded component r2
[cn5:13994] mca: base: components_register: component r2 register function successful
[cn5:13994] mca: base: components_open: opening bml components
[cn5:13994] mca: base: components_open: found loaded component r2
[cn5:13994] mca: base: components_open: component r2 open function successful
[cn2:00789] mca: base: components_register: registering pml components
[cn2:00789] mca: base: components_register: found loaded component v
[cn2:00789] mca: base: components_register: component v register function successful
[cn2:00789] mca: base: components_register: found loaded component bfo
[cn2:00789] mca: base: components_register: component bfo register function successful
[cn2:00789] mca: base: components_register: found loaded component cm
[cn2:00789] mca: base: components_register: component cm register function successful
[cn2:00789] mca: base: components_register: found loaded component ob1
[cn2:00789] mca: base: components_register: component ob1 register function successful
[cn2:00789] mca: base: components_open: opening pml components
[cn2:00789] mca: base: components_open: found loaded component v
[cn4:17657] mca: base: components_register: found loaded component ob1
[cn3:21246] [[15051,1],5]:tcp:recv:handler CONNECTED
[cn3:21246] [[15051,1],5] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 14140 BYTES FOR DEST [[15051,1],5] TAG 20
[cn3:21246] [[15051,1],5] DELIVERING TO RML
[cn3:21246] [[15051,1],5] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21246] [[15051,1],5] message received 14140 bytes from [[15051,0],2] for tag 20
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 464 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 319
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 319 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 245
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 245 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 83 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 464 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 319 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 245 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 83 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 319
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 319 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 107
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 107 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 225
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 225 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_register: registering bml components
[cn5:13995] mca: base: components_register: found loaded component r2
[cn5:13995] mca: base: components_register: component r2 register function successful
[cn5:13995] mca: base: components_open: opening bml components
[cn5:13995] mca: base: components_open: found loaded component r2
[cn5:13995] mca: base: components_open: component r2 open function successful
[cn2:00788] mca: base: components_open: component v open function successful
[cn2:00788] mca: base: components_open: found loaded component bfo
[cn2:00788] mca: base: components_open: component bfo open function successful
[cn2:00788] mca: base: components_open: found loaded component cm
[cn4:17657] mca: base: components_register: component ob1 register function successful
[cn4:17657] mca: base: components_open: opening pml components
[cn4:17657] mca: base: components_open: found loaded component v
[cn3:21246] [[15051,1],5] registering sync ack recvd
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 319 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 107 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 225 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: registering btl components
[cn2:00789] mca: base: components_open: component v open function successful
[cn2:00789] mca: base: components_open: found loaded component bfo
[cn2:00789] mca: base: components_open: component bfo open function successful
[cn2:00789] mca: base: components_open: found loaded component cm
[cn4:17658] mca: base: components_open: component v open function successful
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 986382336
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 986382336
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 175
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 175 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 319
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 319 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 2256
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 2256 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 175 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 319 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 2256 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 104
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 104 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 242
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 242 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 2199
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 2199 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_register: registering btl components
[cn5:13995] mca: base: components_register: found loaded component openib
[cn4:17657] mca: base: components_open: component v open function successful
[cn4:17657] mca: base: components_open: found loaded component bfo
[cn4:17657] mca: base: components_open: component bfo open function successful
[cn4:17657] mca: base: components_open: found loaded component cm
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21246] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 5281349632
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349632
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21246] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 9576316928
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316928
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21246] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 22461218817
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22461218817
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 22461218817
[cn3:21246] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22461218817
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 13871284224
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284224
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21246] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 18166251520
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251520
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21246] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn1:13986] [[15051,0],0] message received 104 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 242 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0] message received 2199 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: found loaded component openib
[cn4:17658] mca: base: components_open: found loaded component bfo
[cn4:17658] mca: base: components_open: component bfo open function successful
[cn4:17658] mca: base: components_open: found loaded component cm
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 18446744070400966657
[cn3:21246] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 18446744070400966657
[cn3:21246] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 986382337
[cn3:21246] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 986382337
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 986382337
[cn3:21246] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 986382337
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 986382337
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 986382337
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 986382337
[cn3:21246] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 986382337
[cn3:21246] fetching data from INTERNAL dstore
[cn3:21246] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 986382337
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 986382337
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 986382337
[cn3:21246] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 986382337
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 5281349633
[cn3:21246] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5281349633
[cn3:21246] storing data in INTERNAL dstore
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 120
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 120 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1800
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1800 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 120 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 1800 bytes from [[15051,0],2] for tag 2
[cn5:13995] mca: base: components_register: component openib register function successful
[cn3:21246] dstore:hash:store storing data for proc 5281349633
[cn3:21246] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5281349633
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 5281349633
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 5281349633
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 5281349633
[cn3:21246] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5281349633
[cn3:21246] fetching data from INTERNAL dstore
[cn3:21246] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 5281349633
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349633
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 5281349633
[cn3:21246] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5281349633
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 9576316929
[cn3:21246] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9576316929
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 9576316929
[cn3:21246] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9576316929
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 9576316929
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 102
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 102 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 2560
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 2560 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 102 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 2560 bytes from [[15051,0],2] for tag 2
[cn5:13995] mca: base: components_register: found loaded component self
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 9576316929
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 9576316929
[cn3:21246] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9576316929
[cn3:21246] fetching data from INTERNAL dstore
[cn3:21246] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 9576316929
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316929
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 9576316929
[cn3:21246] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9576316929
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 13871284225
[cn3:21246] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13871284225
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 13871284225
[cn3:21246] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13871284225
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 13871284225
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 13871284225
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 13871284225
[cn3:21246] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13871284225
[cn3:21246] fetching data from INTERNAL dstore
[cn3:21246] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 13871284225
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284225
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 13871284225
[cn3:21246] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13871284225
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 120
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 120 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 2459
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 2459 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 120 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 2459 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 3276
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 3276 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] mca: base: components_register: component openib register function successful
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 22461218817
[cn3:21246] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 22461218817
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 18166251521
[cn3:21246] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18166251521
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 18166251521
[cn3:21246] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18166251521
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 18166251521
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 18166251521
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 18166251521
[cn3:21246] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18166251521
[cn3:21246] fetching data from INTERNAL dstore
[cn3:21246] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 18166251521
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251521
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 18166251521
[cn3:21246] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18166251521
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 22461218817
[cn3:21246] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22461218817
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 22461218817
[cn3:21246] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22461218817
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 22461218817
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 22461218817
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 118
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 118 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 3276 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] message received 118 bytes from [[15051,0],4] for tag 2
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 22461218817
[cn3:21246] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22461218817
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 26756186113
[cn3:21246] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26756186113
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 26756186113
[cn3:21246] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26756186113
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 26756186113
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 26756186113
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 26756186113
[cn3:21246] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26756186113
[cn3:21246] fetching data from INTERNAL dstore
[cn3:21246] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 26756186113
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26756186113
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 26756186113
[cn3:21246] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26756186113
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 31051153409
[cn3:21246] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31051153409
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 31051153409
[cn3:21246] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31051153409
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 31051153409
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 31051153409
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 31051153409
[cn3:21246] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31051153409
[cn3:21246] fetching data from INTERNAL dstore
[cn3:21246] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 31051153409
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31051153409
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 31051153409
[cn5:13995] mca: base: components_register: component self register function successful
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 100
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 100 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 2443
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 2443 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 100 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 2443 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 116
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 116 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 811
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 811 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_register: found loaded component sm
[cn3:21246] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31051153409
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 35346120705
[cn3:21246] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35346120705
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 35346120705
[cn3:21246] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35346120705
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 35346120705
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 35346120705
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 35346120705
[cn3:21246] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35346120705
[cn3:21246] fetching data from INTERNAL dstore
[cn3:21246] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 35346120705
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35346120705
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 35346120705
[cn3:21246] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35346120705
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 39641088001
[cn3:21246] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39641088001
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 39641088001
[cn3:21246] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39641088001
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 39641088001
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 39641088001
[cn3:21246] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn1:13986] [[15051,0],0] message received 116 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 811 bytes from [[15051,0],2] for tag 2
[cn5:13995] mca: base: components_register: component sm register function successful
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 39641088001
[cn3:21246] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39641088001
[cn3:21246] fetching data from INTERNAL dstore
[cn3:21246] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 39641088001
[cn3:21246] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39641088001
[cn3:21246] storing data in INTERNAL dstore
[cn3:21246] dstore:hash:store storing data for proc 39641088001
[cn3:21246] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39641088001
[cn3:21246] [[15051,1],5] Process bound at launch
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 506
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 506 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 506 bytes from [[15051,0],2] for tag 2
[cn5:13995] mca: base: components_register: found loaded component tcp
[cn3:21246] MCW rank 5 bound to socket 0[core 1[hwt 0]]: [./B/./.][./././.]
[cn3:21246] storing data in NONPEER dstore
[cn3:21246] dstore:hash:store storing data for proc 22461218817
[cn3:21246] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn3:21246] storing data in PEER dstore
[cn3:21246] dstore:hash:store storing data for proc 22461218817
[cn3:21246] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 22461218817
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 102
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 102 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 464 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 102 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 464 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: found loaded component self
[cn3:21245] mca: base: components_register: registering bml components
[cn3:21245] mca: base: components_register: found loaded component r2
[cn3:21245] mca: base: components_register: component r2 register function successful
[cn3:21245] mca: base: components_open: opening bml components
[cn3:21245] mca: base: components_open: found loaded component r2
[cn3:21245] mca: base: components_open: component r2 open function successful
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 190
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 190 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 464 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 190 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 464 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 119
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 119 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 2016
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 2016 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_register: component tcp register function successful
[cn5:13995] mca: base: components_register: found loaded component usnic
[cn3:21246] mca: base: components_register: registering bml components
[cn3:21246] mca: base: components_register: found loaded component r2
[cn3:21246] mca: base: components_register: component r2 register function successful
[cn3:21246] mca: base: components_open: opening bml components
[cn3:21246] mca: base: components_open: found loaded component r2
[cn3:21246] mca: base: components_open: component r2 open function successful
[cn1:13986] [[15051,0],0] message received 119 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 2016 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 188
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 188 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 2016
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 2016 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_register: component usnic register function successful
[cn3:21246] mca: base: components_register: registering btl components
[cn3:21246] mca: base: components_register: found loaded component openib
[cn3:21246] mca: base: components_register: component openib register function successful
[cn3:21246] mca: base: components_register: found loaded component self
[cn3:21246] mca: base: components_register: component self register function successful
[cn3:21246] mca: base: components_register: found loaded component sm
[cn3:21246] mca: base: components_register: component sm register function successful
[cn3:21246] mca: base: components_register: found loaded component tcp
[cn3:21246] mca: base: components_register: component tcp register function successful
[cn3:21246] mca: base: components_register: found loaded component usnic
[cn3:21246] mca: base: components_register: component usnic register function successful
[cn3:21246] mca: base: components_register: found loaded component vader
[cn3:21246] mca: base: components_register: component vader register function successful
[cn3:21246] mca: base: components_open: opening btl components
[cn3:21246] mca: base: components_open: found loaded component openib
[cn3:21246] mca: base: components_open: component openib open function successful
[cn3:21246] mca: base: components_open: found loaded component self
[cn3:21246] mca: base: components_open: component self open function successful
[cn3:21246] mca: base: components_open: found loaded component sm
[cn3:21246] mca: base: components_open: component sm open function successful
[cn3:21246] mca: base: components_open: found loaded component tcp
[cn3:21246] mca: base: components_open: component tcp open function successful
[cn3:21246] mca: base: components_open: found loaded component usnic
[cn3:21246] mca: base: components_open: component usnic open function successful
[cn3:21246] mca: base: components_open: found loaded component vader
[cn3:21246] mca: base: components_open: component vader open function successful
[cn1:13986] [[15051,0],0] message received 188 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 2016 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: component self register function successful
[cn5:13994] mca: base: components_register: found loaded component sm
[cn3:21245] mca: base: components_register: registering btl components
[cn3:21245] mca: base: components_register: found loaded component openib
[cn3:21245] mca: base: components_register: component openib register function successful
[cn3:21245] mca: base: components_register: found loaded component self
[cn3:21245] mca: base: components_register: component self register function successful
[cn3:21245] mca: base: components_register: found loaded component sm
[cn3:21245] mca: base: components_register: component sm register function successful
[cn3:21245] mca: base: components_register: found loaded component tcp
[cn3:21245] mca: base: components_register: component tcp register function successful
[cn3:21245] mca: base: components_register: found loaded component usnic
[cn3:21245] mca: base: components_register: component usnic register function successful
[cn3:21245] mca: base: components_register: found loaded component vader
[cn3:21245] mca: base: components_register: component vader register function successful
[cn3:21245] mca: base: components_open: opening btl components
[cn3:21245] mca: base: components_open: found loaded component openib
[cn3:21245] mca: base: components_open: component openib open function successful
[cn3:21245] mca: base: components_open: found loaded component self
[cn3:21245] mca: base: components_open: component self open function successful
[cn3:21245] mca: base: components_open: found loaded component sm
[cn3:21245] mca: base: components_open: component sm open function successful
[cn3:21245] mca: base: components_open: found loaded component tcp
[cn3:21245] mca: base: components_open: component tcp open function successful
[cn3:21245] mca: base: components_open: found loaded component usnic
[cn3:21245] mca: base: components_open: component usnic open function successful
[cn3:21245] mca: base: components_open: found loaded component vader
[cn3:21245] mca: base: components_open: component vader open function successful
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 103
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 103 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 103 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 116
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 116 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 255
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 255 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_register: found loaded component vader
[cn3:21245] mca: base: components_register: registering pml components
[cn1:13986] [[15051,0],0] message received 116 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 255 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 258
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 258 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] mca: base: components_register: component sm register function successful
[cn3:21246] mca: base: components_register: registering pml components
[cn3:21246] mca: base: components_register: found loaded component v
[cn3:21246] mca: base: components_register: component v register function successful
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 402
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 402 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 258 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] message received 402 bytes from [[15051,0],4] for tag 2
[cn3:21246] mca: base: components_register: found loaded component bfo
[cn3:21246] mca: base: components_register: component bfo register function successful
[cn3:21246] mca: base: components_register: found loaded component cm
[cn5:13995] mca: base: components_register: component vader register function successful
[cn5:13995] mca: base: components_open: opening btl components
[cn5:13995] mca: base: components_open: found loaded component openib
[cn5:13995] mca: base: components_open: component openib open function successful
[cn5:13995] mca: base: components_open: found loaded component self
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 321
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 321 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 187
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 187 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 321 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 187 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_open: component self open function successful
[cn5:13995] mca: base: components_open: found loaded component sm
[cn5:13995] mca: base: components_open: component sm open function successful
[cn5:13995] mca: base: components_open: found loaded component tcp
[cn3:21246] mca: base: components_register: component cm register function successful
[cn3:21246] mca: base: components_register: found loaded component ob1
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 409
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 409 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] message received 409 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 158
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 158 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21246] mca: base: components_register: component ob1 register function successful
[cn5:13995] mca: base: components_open: component tcp open function successful
[cn5:13995] mca: base: components_open: found loaded component usnic
[cn5:13995] mca: base: components_open: component usnic open function successful
[cn5:13995] mca: base: components_open: found loaded component vader
[cn5:13995] mca: base: components_open: component vader open function successful
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 158 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: found loaded component tcp
[cn3:21246] mca: base: components_open: opening pml components
[cn3:21246] mca: base: components_open: found loaded component v
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 190
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 190 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 99
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 99 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 190 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 99 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: component tcp register function successful
[cn5:13994] mca: base: components_register: found loaded component usnic
[cn3:21245] mca: base: components_register: found loaded component v
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 119
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 119 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 186
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 186 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 119 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 186 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: component usnic register function successful
[cn3:21245] mca: base: components_register: component v register function successful
[cn3:21245] mca: base: components_register: found loaded component bfo
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 103
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 103 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 187
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 187 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 103 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 187 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: found loaded component vader
[cn3:21245] mca: base: components_register: component bfo register function successful
[cn3:21245] mca: base: components_register: found loaded component cm
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 182
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 182 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 187
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 187 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 182 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 187 bytes from [[15051,0],2] for tag 2
[cn5:13994] mca: base: components_register: component vader register function successful
[cn5:13994] mca: base: components_open: opening btl components
[cn3:21245] mca: base: components_register: component cm register function successful
[cn3:21245] mca: base: components_register: found loaded component ob1
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 541
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 541 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 541 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 158
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 158 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] mca: base: components_open: found loaded component openib
[cn5:13994] mca: base: components_open: component openib open function successful
[cn5:13994] mca: base: components_open: found loaded component self
[cn5:13994] mca: base: components_open: component self open function successful
[cn5:13994] mca: base: components_open: found loaded component sm
[cn5:13994] mca: base: components_open: component sm open function successful
[cn5:13994] mca: base: components_open: found loaded component tcp
[cn3:21245] mca: base: components_register: component ob1 register function successful
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 409
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 409 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 158 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] message received 409 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 107
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 107 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21245] mca: base: components_open: opening pml components
[cn3:21245] mca: base: components_open: found loaded component v
[cn5:13994] mca: base: components_open: component tcp open function successful
[cn5:13994] mca: base: components_open: found loaded component usnic
[cn5:13994] mca: base: components_open: component usnic open function successful
[cn5:13994] mca: base: components_open: found loaded component vader
[cn5:13994] mca: base: components_open: component vader open function successful
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 107 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],4] for tag 2
[cn3:21246] mca: base: components_open: component v open function successful
[cn5:13994] mca: base: components_register: registering pml components
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 99
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 99 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 319
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 319 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 99 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 319 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 242
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 242 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] mca: base: components_register: found loaded component v
[cn3:21245] mca: base: components_open: component v open function successful
[cn3:21245] mca: base: components_open: found loaded component bfo
[cn3:21245] mca: base: components_open: component bfo open function successful
[cn3:21245] mca: base: components_open: found loaded component cm
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 326
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 326 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 242 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] message received 326 bytes from [[15051,0],4] for tag 2
[cn3:21246] mca: base: components_open: found loaded component bfo
[cn3:21246] mca: base: components_open: component bfo open function successful
[cn3:21246] mca: base: components_open: found loaded component cm
[cn5:13995] mca: base: components_register: registering pml components
[cn5:13995] mca: base: components_register: found loaded component v
[cn5:13995] mca: base: components_register: component v register function successful
[cn5:13995] mca: base: components_register: found loaded component bfo
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],4] for tag 2
[cn5:13995] mca: base: components_register: component bfo register function successful
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 186
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 186 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 186 bytes from [[15051,0],4] for tag 2
[cn5:13995] mca: base: components_register: found loaded component cm
[cn5:13995] mca: base: components_register: component cm register function successful
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],4] for tag 2
[cn5:13995] mca: base: components_register: found loaded component ob1
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 245
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 245 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 245 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 115
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 115 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_register: component ob1 register function successful
[cn5:13995] mca: base: components_open: opening pml components
[cn5:13995] mca: base: components_open: found loaded component v
[cn1:13986] [[15051,0],0] message received 115 bytes from [[15051,0],4] for tag 2
[cn5:13994] mca: base: components_register: component v register function successful
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],4] for tag 2
[cn5:13994] mca: base: components_register: found loaded component bfo
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 187
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 187 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 187 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 187
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 187 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] mca: base: components_register: component bfo register function successful
[cn5:13994] mca: base: components_register: found loaded component cm
[cn1:13986] [[15051,0],0] message received 187 bytes from [[15051,0],4] for tag 2
[cn5:13994] mca: base: components_register: component cm register function successful
[cn5:13994] mca: base: components_register: found loaded component ob1
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 180
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 180 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 180 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 95
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 95 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] mca: base: components_register: component ob1 register function successful
[cn5:13994] mca: base: components_open: opening pml components
[cn1:13986] [[15051,0],0] message received 95 bytes from [[15051,0],4] for tag 2
[cn5:13994] mca: base: components_open: found loaded component v
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 107
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 107 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 107 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 319
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 319 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] mca: base: components_open: component v open function successful
[cn1:13986] [[15051,0],0] message received 319 bytes from [[15051,0],4] for tag 2
[cn5:13994] mca: base: components_open: component v open function successful
[cn5:13994] mca: base: components_open: found loaded component bfo
[cn5:13994] mca: base: components_open: component bfo open function successful
[cn5:13994] mca: base: components_open: found loaded component cm
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 242
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 242 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 242 bytes from [[15051,0],4] for tag 2
[cn5:13995] mca: base: components_open: found loaded component bfo
[cn5:13995] mca: base: components_open: component bfo open function successful
[cn5:13995] mca: base: components_open: found loaded component cm
[cn1:13986] [[15051,0],0] plm:base:launch wiring up iof for job [15051,1]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 172
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 172 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 172 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13994] mca: base: components_register: registering mtl components
[cn5:13994] mca: base: components_register: found loaded component psm
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 352
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 172 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 352 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 352 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13994] mca: base: components_register: component psm register function successful
[cn5:13994] mca: base: components_open: opening mtl components
[cn5:13994] mca: base: components_open: found loaded component psm
[cn5:13994] mca: base: close: component psm closed
[cn5:13994] mca: base: close: unloading component psm
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 279
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 352 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 279 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 279 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] message received 279 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13994] mca: base: close: component cm closed
[cn5:13994] mca: base: close: unloading component cm
[cn5:13994] mca: base: components_open: found loaded component ob1
[cn5:13994] mca: base: components_open: component ob1 open function successful
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 172 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 172
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 172 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] message received 172 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13995] mca: base: components_register: registering mtl components
[cn5:13995] mca: base: components_register: found loaded component psm
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 247 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 247
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 247 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 238 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 247 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13995] mca: base: components_register: component psm register function successful
[cn5:13995] mca: base: components_open: opening mtl components
[cn5:13995] mca: base: components_open: found loaded component psm
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 176 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 238
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 238 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 238 bytes from [[15051,0],4] for tag 2
[cn5:13995] mca: base: close: component psm closed
[cn5:13995] mca: base: close: unloading component psm
[cn5:13995] mca: base: close: component cm closed
[cn5:13995] mca: base: close: unloading component cm
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 176
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 176 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 176 bytes from [[15051,0],4] for tag 2
[cn5:13995] mca: base: components_open: found loaded component ob1
[cn5:13995] mca: base: components_open: component ob1 open function successful
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],1] for tag 2
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 259 BYTES ON SOCKET 9
[cn2:00788] mca: base: components_register: registering mtl components
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 259
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 259 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 259 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 265
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 265 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00789] mca: base: components_register: registering mtl components
[cn2:00789] mca: base: components_register: found loaded component psm
[cn2:00789] mca: base: components_register: component psm register function successful
[cn1:13986] [[15051,0],0] message received 265 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 279
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 279 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 265 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00789] mca: base: components_open: opening mtl components
[cn2:00789] mca: base: components_open: found loaded component psm
[cn2:00789] mca: base: close: component psm closed
[cn2:00789] mca: base: close: unloading component psm
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 279 bytes from [[15051,0],1] for tag 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 279 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 101 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00789] mca: base: close: component cm closed
[cn2:00789] mca: base: close: unloading component cm
[cn2:00789] mca: base: components_open: found loaded component ob1
[cn2:00789] mca: base: components_open: component ob1 open function successful
[cn1:13986] [[15051,0],0] message received 101 bytes from [[15051,0],1] for tag 2
[cn2:00788] mca: base: components_register: found loaded component psm
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 352
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 352 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 101 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] message received 352 bytes from [[15051,0],1] for tag 2
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00788] mca: base: components_register: component psm register function successful
[cn2:00788] mca: base: components_open: opening mtl components
[cn2:00788] mca: base: components_open: found loaded component psm
[cn2:00788] mca: base: close: component psm closed
[cn2:00788] mca: base: close: unloading component psm
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 133
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 133 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 133 bytes from [[15051,0],1] for tag 2
[cn2:00788] mca: base: close: component cm closed
[cn2:00788] mca: base: close: unloading component cm
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 176
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 176 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 176 bytes from [[15051,0],1] for tag 2
[cn2:00788] mca: base: components_open: found loaded component ob1
[cn2:00788] mca: base: components_open: component ob1 open function successful
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 352 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 133 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 172
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 176 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 172 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 172 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 172 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21245] mca: base: components_register: registering mtl components
[cn3:21245] mca: base: components_register: found loaded component psm
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 352 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 352
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 352 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0] message received 352 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21245] mca: base: components_register: component psm register function successful
[cn3:21245] mca: base: components_open: opening mtl components
[cn3:21245] mca: base: components_open: found loaded component psm
[cn3:21245] mca: base: close: component psm closed
[cn3:21245] mca: base: close: unloading component psm
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 133 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 133
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 133 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 133 bytes from [[15051,0],2] for tag 2
[cn3:21245] mca: base: close: component cm closed
[cn3:21245] mca: base: close: unloading component cm
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 176
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 176 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 176 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 176 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 172
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 172 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 172 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21245] mca: base: components_open: found loaded component ob1
[cn3:21245] mca: base: components_open: component ob1 open function successful
[cn1:13986] [[15051,0],0] message received 172 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21246] mca: base: components_register: registering mtl components
[cn3:21246] mca: base: components_register: found loaded component psm
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 298 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 187 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 298
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 176 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 298 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 298 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 187
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 187 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21246] mca: base: components_register: component psm register function successful
[cn3:21246] mca: base: components_open: opening mtl components
[cn3:21246] mca: base: components_open: found loaded component psm
[cn3:21246] mca: base: close: component psm closed
[cn1:13986] [[15051,0],0] message received 187 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 176
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 176 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21246] mca: base: close: unloading component psm
[cn3:21246] mca: base: close: component cm closed
[cn3:21246] mca: base: close: unloading component cm
[cn1:13986] [[15051,0],0] message received 176 bytes from [[15051,0],2] for tag 2
[cn3:21246] mca: base: components_open: found loaded component ob1
[cn3:21246] mca: base: components_open: component ob1 open function successful
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 172
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 172 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 172 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 352 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] message received 172 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17657] mca: base: components_register: registering mtl components
[cn4:17657] mca: base: components_register: found loaded component psm
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 352
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 352 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 352 bytes from [[15051,0],3] for tag 2
[cn4:17657] mca: base: components_register: component psm register function successful
[cn4:17657] mca: base: components_open: opening mtl components
[cn4:17657] mca: base: components_open: found loaded component psm
[cn4:17657] mca: base: close: component psm closed
[cn4:17657] mca: base: close: unloading component psm
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 279
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 279 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 279 bytes from [[15051,0],3] for tag 2
[cn4:17657] mca: base: close: component cm closed
[cn4:17657] mca: base: close: unloading component cm
[cn4:17657] mca: base: components_open: found loaded component ob1
[cn4:17657] mca: base: components_open: component ob1 open function successful
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 172
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 172 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 172 bytes from [[15051,0],3] for tag 2
[cn4:17658] mca: base: components_register: registering mtl components
[cn4:17658] mca: base: components_register: found loaded component psm
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 279 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 172 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17658] mca: base: components_register: component psm register function successful
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 514
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 514 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 514 bytes from [[15051,0],3] for tag 2
[cn4:17658] mca: base: components_open: opening mtl components
[cn4:17658] mca: base: components_open: found loaded component psm
[cn4:17658] mca: base: close: component psm closed
[cn4:17658] mca: base: close: unloading component psm
[cn4:17658] mca: base: close: component cm closed
[cn4:17658] mca: base: close: unloading component cm
[cn4:17658] mca: base: components_open: found loaded component ob1
[cn4:17658] mca: base: components_open: component ob1 open function successful
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 514 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 86
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 86 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 86 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 86 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 249 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13994] select: component v not in the include list
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 249
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 249 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 249 bytes from [[15051,0],4] for tag 2
[cn5:13995] select: component v not in the include list
[cn5:13995] select: component bfo not in the include list
[cn5:13995] select: initializing pml component ob1
[cn5:13995] select: initializing btl component openib
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 139
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 139 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 139 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 139 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 84 BYTES ON SOCKET 9
[cn5:13994] select: component bfo not in the include list
[cn5:13994] select: initializing pml component ob1
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 84
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 84 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 84 bytes from [[15051,0],4] for tag 2
[cn5:13994] select: initializing btl component openib
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,1],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,1],0] (ORIGIN [[15051,1],0]) OF 74 BYTES FOR DEST [[15051,0],0] TAG 1
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 74 bytes from [[15051,1],0] for tag 1
[cn1:13986] [[15051,0],0] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn1:13986] [[15051,0],0] orted_recv: received sync+nidmap from local proc [[15051,1],0]
[cn1:13986] [[15051,0],0] rml_send_buffer to peer [[15051,1],0] at tag 20
[cn1:13986] [[15051,0],0] rml_send_msg to peer [[15051,1],0] at tag 20
[cn1:13986] [[15051,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] oob:base:send to target [[15051,1],0]
[cn1:13986] [[15051,0],0] oob:base:send known transport for peer [[15051,1],0]
[cn1:13986] [[15051,0],0] oob:tcp:send_nb to peer [[15051,1],0]:20
[cn1:13986] [[15051,0],0] tcp:send_nb to peer [[15051,1],0]
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,1],0]
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,1],0]:20
[cn1:13986] [[15051,0],0] routed_binomial_get([[15051,1],0]) --> [[15051,1],0]
[cn1:13986] [[15051,0],0] tcp:send_nb: already connected to [[15051,1],0] - queueing for send
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,1],0]
[cn1:13986] [[15051,0],0] tcp:send_handler called to send to peer [[15051,1],0]
[cn1:13986] [[15051,0],0] tcp:send_handler SENDING TO [[15051,1],0]
[cn1:13986] [[15051,0],0] MESSAGE SEND COMPLETE TO [[15051,1],0] OF 14140 BYTES ON SOCKET 20
[cn1:13986] [[15051,0],0]-[[15051,1],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14001] [[15051,1],0]:tcp:recv:handler called for peer [[15051,0],0]
[cn1:14001] [[15051,1],0]:tcp:recv:handler CONNECTED
[cn1:14001] [[15051,1],0]:tcp:recv:handler allocate new recv msg
[cn1:14001] [[15051,1],0]:tcp:recv:handler read hdr
[cn1:14001] [[15051,1],0]:tcp:recv:handler allocate data region of size 14140
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 86
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 86 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 86 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 86 bytes from [[15051,0],1] for tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 249 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00789] select: component v not in the include list
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 249
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 249 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 249 bytes from [[15051,0],1] for tag 2
[cn2:00788] select: component v not in the include list
[cn2:00788] select: component bfo not in the include list
[cn2:00788] select: initializing pml component ob1
[cn2:00788] select: initializing btl component openib
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 139
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 139 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 139 bytes from [[15051,0],1] for tag 2
[cn2:00789] select: component bfo not in the include list
[cn2:00789] select: initializing pml component ob1
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 139 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 84
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 84 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 84 bytes from [[15051,0],1] for tag 2
[cn2:00789] select: initializing btl component openib
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 84 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 86
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 86 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 86 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 86 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 249 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21245] select: component v not in the include list
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 249
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 249 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 249 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 88 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 88
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 88 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21246] select: component v not in the include list
[cn3:21246] select: component bfo not in the include list
[cn3:21246] select: initializing pml component ob1
[cn3:21246] select: initializing btl component openib
[cn1:13986] [[15051,0],0] message received 88 bytes from [[15051,0],2] for tag 2
[cn3:21245] select: component bfo not in the include list
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 81
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 81 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 81 bytes from [[15051,0],2] for tag 2
[cn3:21245] select: initializing pml component ob1
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 81 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 84 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 84
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 84 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 84 bytes from [[15051,0],2] for tag 2
[cn3:21245] select: initializing btl component openib
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 86
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 86 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 249 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 86 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 139 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] message received 86 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 249
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 249 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17658] select: component v not in the include list
[cn1:13986] [[15051,0],0] message received 249 bytes from [[15051,0],3] for tag 2
[cn4:17657] select: component v not in the include list
[cn4:17657] select: component bfo not in the include list
[cn4:17657] select: initializing pml component ob1
[cn4:17657] select: initializing btl component openib
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 139
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 139 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 139 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17658] select: component bfo not in the include list
[cn4:17658] select: initializing pml component ob1
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 84 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 84
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 84 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 84 bytes from [[15051,0],3] for tag 2
[cn4:17658] select: initializing btl component openib
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 194
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 194 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 194 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 194 bytes from [[15051,0],4] for tag 2
[cn5][[15051,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 219
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 219 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 219 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 219 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 215 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1620 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 215
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 215 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5][[15051,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn1:13986] [[15051,0],0] message received 215 bytes from [[15051,0],4] for tag 2
[cn5:13994] [rank=8] openib: using device mlx4_0
[cn5:13994] openib BTL: rdmacm IP address not found on port
[cn5:13994] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1620
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1620 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 1620 bytes from [[15051,0],4] for tag 2
[cn5][[15051,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn5][[15051,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn5][[15051,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn5][[15051,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn5][[15051,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn5][[15051,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn5][[15051,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn5][[15051,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x18f6f60 for btl 0x1904c60
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 216
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 216 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 216 bytes from [[15051,0],4] for tag 2
[cn5][[15051,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 216 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,1],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,1],1] (ORIGIN [[15051,1],1]) OF 74 BYTES FOR DEST [[15051,0],0] TAG 1
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 74 bytes from [[15051,1],1] for tag 1
[cn1:13986] [[15051,0],0] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn1:13986] [[15051,0],0] orted_recv: received sync+nidmap from local proc [[15051,1],1]
[cn1:13986] [[15051,0],0] rml_send_buffer to peer [[15051,1],1] at tag 20
[cn1:13986] [[15051,0],0] rml_send_msg to peer [[15051,1],1] at tag 20
[cn1:13986] [[15051,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] oob:base:send to target [[15051,1],1]
[cn1:13986] [[15051,0],0] oob:base:send known transport for peer [[15051,1],1]
[cn1:13986] [[15051,0],0] oob:tcp:send_nb to peer [[15051,1],1]:20
[cn1:13986] [[15051,0],0] tcp:send_nb to peer [[15051,1],1]
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,1],1]
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,1],1]:20
[cn1:13986] [[15051,0],0] routed_binomial_get([[15051,1],1]) --> [[15051,1],1]
[cn1:13986] [[15051,0],0] tcp:send_nb: already connected to [[15051,1],1] - queueing for send
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,1],1]
[cn1:13986] [[15051,0],0] tcp:send_handler called to send to peer [[15051,1],1]
[cn1:13986] [[15051,0],0] tcp:send_handler SENDING TO [[15051,1],1]
[cn1:13986] [[15051,0],0] MESSAGE SEND COMPLETE TO [[15051,1],1] OF 14140 BYTES ON SOCKET 23
[cn1:13986] [[15051,0],0]-[[15051,1],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] plm:base:launch [15051,1] registered
[cn1:13986] [[15051,0],0] plm:base:launch job [15051,1] is not a dynamic spawn
[cn1:13986] [[15051,0],0]: Setting up debugger process table for applications
  MPIR_being_debugged = 0
  MPIR_debug_state = 1
  MPIR_partial_attach_ok = 1
  MPIR_i_am_starter = 0
  MPIR_forward_output = 0
  MPIR_proctable_size = 10
  MPIR_proctable:
    (i, host, exe, pid) = (0, cn1, /home/research/artpol/ompi_dev/app/./hellompi, 14001)
    (i, host, exe, pid) = (1, cn1, /home/research/artpol/ompi_dev/app/./hellompi, 14002)
    (i, host, exe, pid) = (2, cn2, /home/research/artpol/ompi_dev/app/./hellompi, 788)
    (i, host, exe, pid) = (3, cn2, /home/research/artpol/ompi_dev/app/./hellompi, 789)
    (i, host, exe, pid) = (4, cn3, /home/research/artpol/ompi_dev/app/./hellompi, 21245)
    (i, host, exe, pid) = (5, cn3, /home/research/artpol/ompi_dev/app/./hellompi, 21246)
    (i, host, exe, pid) = (6, cn4, /home/research/artpol/ompi_dev/app/./hellompi, 17657)
    (i, host, exe, pid) = (7, cn4, /home/research/artpol/ompi_dev/app/./hellompi, 17658)
    (i, host, exe, pid) = (8, cn5, /home/research/artpol/ompi_dev/app/./hellompi, 13994)
    (i, host, exe, pid) = (9, cn5, /home/research/artpol/ompi_dev/app/./hellompi, 13995)
MPIR_executable_path: NULL
MPIR_server_arguments: NULL
[cn1:14002] [[15051,1],1]:tcp:recv:handler called for peer [[15051,0],0]
[cn1:14002] [[15051,1],1]:tcp:recv:handler CONNECTED
[cn1:14002] [[15051,1],1]:tcp:recv:handler allocate new recv msg
[cn1:14002] [[15051,1],1]:tcp:recv:handler read hdr
[cn1:14002] [[15051,1],1]:tcp:recv:handler allocate data region of size 14140
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 194
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 194 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 194 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 194 bytes from [[15051,0],1] for tag 2
[cn2][[15051,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 219
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 219 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 219 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 219 bytes from [[15051,0],1] for tag 2
[cn2][[15051,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 79
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 79 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 79 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1428
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 79 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 194 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 1428 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 194
[cn2:00789] [rank=3] openib: using device mlx4_0
[cn1:13986] [[15051,0],0] message received 1428 bytes from [[15051,0],1] for tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1428 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 166 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 220 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2][[15051,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn2][[15051,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn2][[15051,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn2][[15051,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn2][[15051,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn2][[15051,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn2][[15051,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 194
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 166
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 166 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 194 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 166 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 194 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 194 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 219 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 79 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1428 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 215 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00789] openib BTL: rdmacm IP address not found on port
[cn2:00789] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn3][[15051,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 220
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 220 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 219
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 219 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 220 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 219 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 216
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 79
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 79 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1809 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2][[15051,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x818fa0 for btl 0x826ca0
[cn3][[15051,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 194 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 79 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0] message received 194 bytes from [[15051,0],4] for tag 2
[cn3:21245] [rank=4] openib: using device mlx4_0
[cn5][[15051,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 215
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 215 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 166 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 220 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 216 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1428
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1428 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 215 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0] message received 216 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 1428 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1809
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1809 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 216 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 166
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 166 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13995] [rank=9] openib: using device mlx4_0
[cn5:13995] openib BTL: rdmacm IP address not found on port
[cn5:13995] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn2][[15051,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn3][[15051,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn3][[15051,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn3][[15051,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn3][[15051,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn3][[15051,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn3][[15051,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn3][[15051,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn1:13986] [[15051,0],0] message received 1809 bytes from [[15051,0],4] for tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] message received 166 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5][[15051,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn5][[15051,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn5][[15051,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn5][[15051,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn5][[15051,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn5][[15051,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn5][[15051,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn5][[15051,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn5][[15051,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x266cf80 for btl 0x267ac80
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21245] openib BTL: rdmacm IP address not found on port
[cn3:21245] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 220
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 216 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 220 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 220 bytes from [[15051,0],2] for tag 2
[cn3][[15051,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0xbe4f00 for btl 0xbf2a60
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 216
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 216 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 216 bytes from [[15051,0],2] for tag 2
[cn3][[15051,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 216
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 216 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 216 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 216 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5][[15051,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1047
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1047 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1047 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 1047 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 596 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 162 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 596
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 596 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] storing data in PEER dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 35346120705
[cn5:13994] select: init of component openib returned success
[cn5:13994] select: initializing btl component self
[cn5:13994] select: init of component self returned success
[cn5:13994] select: initializing btl component sm
[cn5:13994] select: init of component sm returned success
[cn5:13994] select: initializing btl component tcp
[cn5:13994] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn5:13994] btl: tcp: Found match: 127.0.0.1 (lo)
[cn5:13994] storing data in PEER dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 35346120705
[cn5:13994] select: init of component tcp returned success
[cn5:13994] select: initializing btl component usnic
[cn5:13994] usNIC connectivity agent initialized
[cn1:13986] [[15051,0],0] message received 596 bytes from [[15051,0],4] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 162
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 162 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5][[15051,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 8, qpn 27531
[cn5][[15051,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 8, Port: 1, QPN: 27531
[cn5][[15051,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn1:13986] [[15051,0],0] message received 162 bytes from [[15051,0],4] for tag 2
[cn5:13994] usNIC connectivity client initialized
[cn5:13994] found 1 verbs interface
[cn5:13994] examining verbs interface: mlx4_0
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 412
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 412 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 412 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] message received 412 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13994] found 1 verbs interface
[cn5:13994] examining verbs interface: mlx4_0
[cn5:13994] btl:usnic: no usNICs found
[cn5:13994] storing data in PEER dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 35346120705
[cn5:13994] select: init of component usnic returned failure
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 596
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 596 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 596 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 596 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5][[15051,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 8, qpn 27532
[cn5][[15051,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 8, Port: 1, QPN: 27532
[cn5][[15051,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 998 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 998
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 240 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 998 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 998 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 230 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 240
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 240 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13995] storing data in PEER dstore
[cn5:13995] dstore:hash:store storing data for proc 39641088001
[cn5:13995] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 39641088001
[cn5:13995] select: init of component openib returned success
[cn5:13995] select: initializing btl component self
[cn5:13995] select: init of component self returned success
[cn5:13995] select: initializing btl component sm
[cn5:13995] select: init of component sm returned success
[cn5:13995] select: initializing btl component tcp
[cn5:13995] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn5:13995] btl: tcp: Found match: 127.0.0.1 (lo)
[cn5:13995] storing data in PEER dstore
[cn5:13995] dstore:hash:store storing data for proc 39641088001
[cn5:13995] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 39641088001
[cn5:13995] select: init of component tcp returned success
[cn5:13995] select: initializing btl component usnic
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 240 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 618 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13994] usNIC connectivity client finalized
[cn5:13994] mca: base: close: component usnic closed
[cn5:13994] mca: base: close: unloading component usnic
[cn5:13994] select: initializing btl component vader
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 230
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 230 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 230 bytes from [[15051,0],4] for tag 2
[cn5:13994] storing data in PEER dstore
[cn5:13994] dstore:hash:store storing data for proc 35346120705
[cn5:13994] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 35346120705
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 618
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 618 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 618 bytes from [[15051,0],4] for tag 2
[cn5:13994] select: init of component vader returned success
[cn5:13994] select: init returned priority 20
[cn5:13994] selected ob1 best priority 20
[cn5:13994] select: component ob1 selected
[cn5:13994] mca: base: close: component v closed
[cn5:13994] mca: base: close: unloading component v
[cn5:13994] mca: base: close: component bfo closed
[cn5:13994] mca: base: close: unloading component bfo
[cn5:13994] [[15051,1],8] running modex
[cn5:13994] [[15051,1],8] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 206
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 206 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 206 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 206 bytes from [[15051,0],4] for tag 2
[cn5:13995] [[15051,1],9] ORTE_ERROR_LOG: Connection reset by peer in file /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/usnic/btl_usnic_cclient.c at line 124
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 194
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 194
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 194 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 194 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 194 bytes from [[15051,0],1] for tag 2
[cn2][[15051,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 194 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 161
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 194 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 194 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 219
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 219 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3][[15051,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn1:13986] [[15051,0],0] message received 219 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 161 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3][[15051,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 161 bytes from [[15051,0],1] for tag 2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2][[15051,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] [cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 219 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 161 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 216
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 216 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 79
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 79 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 79 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 216 bytes from [[15051,0],1] for tag 2
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 79 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 877
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 216 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 877 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1005
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1005 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn2][[15051,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] [cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21246] [rank=5] openib: using device mlx4_0
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 877 bytes from [[15051,0],1] for tag 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 1005 bytes from [[15051,0],2] for tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 877 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn2][[15051,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn2][[15051,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn2][[15051,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn2][[15051,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn3][[15051,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn3][[15051,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn3][[15051,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn3][[15051,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn3][[15051,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 215
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1005 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 215 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 215 bytes from [[15051,0],1] for tag 2
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn2:00788] [rank=2] openib: using device mlx4_0
[cn2:00788] openib BTL: rdmacm IP address not found on port
[cn2:00788] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 645
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 645 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 643
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 643 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 645 bytes from [[15051,0],1] for tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 215 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 643 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 166
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 166 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2][[15051,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn2][[15051,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn2][[15051,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x1d22fa0 for btl 0x1d2e460
[cn3][[15051,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn3][[15051,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn3][[15051,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0xa79f00 for btl 0xa87a60
[cn1:13986] [[15051,0],0] message received 166 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 643 BYTES ON SOCKET 9
[cn3:21246] openib BTL: rdmacm IP address not found on port
[cn3:21246] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 166 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 645 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 194
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 564
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1125
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1125 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 194 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1125 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 194 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] message received 1125 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4][[15051,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13994] [[15051,1],8] grpcomm:base:modex: performing modex
[cn5:13994] fetching data from PEER dstore
[cn5:13994] dstore:hash:fetch: searching for key NULL on proc 35346120705
[cn5:13994] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 35346120705
[cn5:13994] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 35346120705
[cn5:13994] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 35346120705
[cn5:13994] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 35346120705
[cn5:13994] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 35346120705
[cn5:13994] [[15051,1],8] grpcomm:base:pack_modex: reporting 5 entries
[cn5:13994] [[15051,1],8] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn5:13994] [[15051,1],8] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn5:13994] [[15051,1],8] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn5:13994] [[15051,1],8] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn5:13994] [[15051,1],8] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1493
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1493 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 564 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 564 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0] message received 1493 bytes from [[15051,0],4] for tag 2
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 194 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] message received 564 bytes from [[15051,0],2] for tag 2
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,1],8]
[cn5:13991] [[15051,0],4]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 596
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 596 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4]:tcp:recv:handler allocate data region of size 348
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 219
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 219 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 216
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 216 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 216 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13994] [[15051,1],8] grpcomm:base:full:modex: executing allgather
[cn5:13994] [[15051,1],8] grpcomm:bad entering allgather
[cn5:13994] [[15051,1],8] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn5:13994] [[15051,1],8] grpcomm:base:modex: modex posted
[cn5:13994] [[15051,1],8] grpcomm:bad sending collective 0 to our daemon
[cn5:13994] [[15051,1],8] rml_send_buffer to peer [[15051,0],4] at tag 30
[cn5:13994] [[15051,1],8] grpcomm:bad allgather underway
[cn5:13994] [[15051,1],8] rml_send_msg to peer [[15051,0],4] at tag 30
[cn5:13994] [[15051,1],8] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13994] [[15051,1],8] oob:base:send to target [[15051,0],4]
[cn5:13994] [[15051,1],8] oob:base:send known transport for peer [[15051,0],4]
[cn5:13994] [[15051,1],8] oob:tcp:send_nb to peer [[15051,0],4]:30
[cn5:13994] [[15051,1],8] tcp:send_nb to peer [[15051,0],4]
[cn5:13994] [[15051,1],8]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],4]
[cn5:13994] [[15051,1],8]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],4]:30
[cn5:13994] [[15051,1],8] routed_binomial_get([[15051,0],4]) --> [[15051,0],4]
[cn5:13994] [[15051,1],8] tcp:send_nb: already connected to [[15051,0],4] - queueing for send
[cn3:21245] storing data in PEER dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 18166251521
[cn3:21245] select: init of component openib returned success
[cn3:21245] select: initializing btl component self
[cn3:21245] select: init of component self returned success
[cn3:21245] select: initializing btl component sm
[cn3:21245] select: init of component sm returned success
[cn3:21245] select: initializing btl component tcp
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 596 bytes from [[15051,0],1] for tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 219 bytes from [[15051,0],3] for tag 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] message received 216 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 596 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2][[15051,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 4, qpn 37950
[cn2][[15051,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 4, Port: 1, QPN: 37950
[cn2][[15051,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn4][[15051,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn3][[15051,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 561
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 561 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 998
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 998 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 79
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 79 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 464 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1493 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 596
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 596 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 561 bytes from [[15051,0],4] for tag 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 998 bytes from [[15051,0],1] for tag 2
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] message received 79 bytes from [[15051,0],3] for tag 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 596 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 828
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 828 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 464
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 464 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 596 BYTES ON SOCKET 9
[cn5:13994] [[15051,1],8]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],4]
[cn5:13994] [[15051,1],8] tcp:send_handler called to send to peer [[15051,0],4]
[cn5:13994] [[15051,1],8] tcp:send_handler SENDING TO [[15051,0],4]
[cn5:13994] [[15051,1],8] MESSAGE SEND COMPLETE TO [[15051,0],4] OF 348 BYTES ON SOCKET 11
[cn5:13994] [[15051,1],8]-[[15051,0],4] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00789] storing data in PEER dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 13871284225
[cn2:00789] select: init of component openib returned success
[cn2:00789] select: initializing btl component self
[cn2:00789] select: init of component self returned success
[cn2:00789] select: initializing btl component sm
[cn2:00789] select: init of component sm returned success
[cn2:00789] select: initializing btl component tcp
[cn2:00789] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn2:00789] btl: tcp: Found match: 127.0.0.1 (lo)
[cn2:00789] storing data in PEER dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 13871284225
[cn2:00789] select: init of component tcp returned success
[cn2:00789] select: initializing btl component usnic
[cn4:17657] [rank=6] openib: using device mlx4_0
[cn3][[15051,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 7, qpn 33158
[cn3][[15051,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 7, Port: 1, QPN: 33158
[cn3][[15051,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 104
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 79 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 104 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 561 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 216
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 216 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] message received 828 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 464 bytes from [[15051,0],2] for tag 2
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 219 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] message received 104 bytes from [[15051,0],4] for tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 998 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] message received 216 bytes from [[15051,0],1] for tag 2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn4][[15051,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn4][[15051,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn4][[15051,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn4][[15051,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 216 BYTES ON SOCKET 9
[cn3:21245] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn3:21245] btl: tcp: Found match: 127.0.0.1 (lo)
[cn3:21245] storing data in PEER dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 18166251521
[cn3:21245] select: init of component tcp returned success
[cn3:21245] select: initializing btl component usnic
[cn5:13995] [[15051,1],9] rml_send_buffer to peer [[15051,0],0] at tag 34
[cn2][[15051,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1793
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 1793 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 80 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 233
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 79 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 112 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 166
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 166 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 79
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 828 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 79 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 1793 bytes from [[15051,0],4] for tag 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 104 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] message received 166 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 166 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 79 bytes from [[15051,0],2] for tag 2
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 233 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,1],9]
[cn5:13991] [[15051,0],4]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 820 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 233 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 820
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 820 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4]:tcp:recv:handler allocate data region of size 700
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 80
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 80 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13995] [[15051,1],9] rml_send_msg to peer [[15051,0],0] at tag 34
[cn5:13995] [[15051,1],9] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13995] [[15051,1],9] oob:base:send to target [[15051,0],0]
[cn5:13995] [[15051,1],9] oob:base:send unknown peer [[15051,0],0]
[cn5:13995] fetching data from PEER dstore
[cn5:13995] dstore:hash:fetch: searching for key orte.rmluri on proc 986382336
[cn5:13995] dstore_hash:fetch key orte.rmluri for proc 986382336 not found
[cn5:13995] [[15051,1],9] routed_binomial_get([[15051,0],0]) --> [[15051,0],4]
[cn5:13995] [[15051,1],9] is reachable by TCP
[cn5:13995] [[15051,1],9] oob:tcp:send_nb to peer [[15051,0],0]:34
[cn5:13995] [[15051,1],9] tcp:send_nb to peer [[15051,0],0]
[cn5:13995] [[15051,1],9]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13995] [[15051,1],9]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:34
[cn5:13995] [[15051,1],9] routed_binomial_get([[15051,0],0]) --> [[15051,0],4]
[cn5:13995] [[15051,1],9] tcp:send_nb: already connected to [[15051,0],4] - queueing for send
[cn5:13995] [[15051,1],9]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13995] [[15051,1],9] tcp:send_handler called to send to peer [[15051,0],4]
[cn5:13995] [[15051,1],9] tcp:send_handler SENDING TO [[15051,0],4]
[cn5:13995] [[15051,1],9] MESSAGE SEND COMPLETE TO [[15051,0],4] OF 700 BYTES ON SOCKET 11
[cn5:13995] [[15051,1],9]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17657] openib BTL: rdmacm IP address not found on port
[cn4:17657] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn3:21245] usNIC connectivity agent initialized
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 214 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] message received 233 bytes from [[15051,0],1] for tag 2
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 820 bytes from [[15051,0],3] for tag 2
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] message received 80 bytes from [[15051,0],2] for tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 93
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 214
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 214 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 216
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 209 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 112
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 112 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 453 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2][[15051,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 4, qpn 37951
[cn4][[15051,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn4][[15051,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn4][[15051,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn4][[15051,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x917fc0 for btl 0x923460
[cn3:21245] usNIC connectivity client initialized
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] message received 214 bytes from [[15051,0],1] for tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] message received 112 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn2][[15051,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 4, Port: 1, QPN: 37951
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21245] found 1 verbs interface
[cn3:21245] examining verbs interface: mlx4_0
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 209
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 209 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 998
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 998 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 209 bytes from [[15051,0],1] for tag 2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 998 bytes from [[15051,0],2] for tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1793 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 139 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 596
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 596 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2][[15051,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21246] storing data in PEER dstore
[cn3:21246] dstore:hash:store storing data for proc 22461218817
[cn3:21246] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 22461218817
[cn3:21246] select: init of component openib returned success
[cn3:21246] select: initializing btl component self
[cn3:21246] select: init of component self returned success
[cn3:21246] select: initializing btl component sm
[cn3:21246] select: init of component sm returned success
[cn3:21246] select: initializing btl component tcp
[cn3:21246] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn3:21246] btl: tcp: Found match: 127.0.0.1 (lo)
[cn3:21246] storing data in PEER dstore
[cn3:21246] dstore:hash:store storing data for proc 22461218817
[cn3:21246] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 22461218817
[cn3:21246] select: init of component tcp returned success
[cn3:21246] select: initializing btl component usnic
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 93 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 2
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 453
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 104 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 453 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 216 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0] message received 596 bytes from [[15051,0],2] for tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 998 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] message received 93 bytes from [[15051,0],4] for tag 2
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 453 bytes from [[15051,0],1] for tag 2
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 216 bytes from [[15051,0],3] for tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3][[15051,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 7, qpn 33159
[cn3][[15051,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 7, Port: 1, QPN: 33159
[cn3][[15051,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn5:13995] sess_dir_finalize: proc session dir does not exist
[cn2:00788] storing data in PEER dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 9576316929
[cn2:00788] select: init of component openib returned success
[cn2:00788] select: initializing btl component self
[cn2:00788] select: init of component self returned success
[cn2:00788] select: initializing btl component sm
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 93 BYTES ON SOCKET 9
[cn4][[15051,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 139
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 139 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 460
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 460 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 139 bytes from [[15051,0],1] for tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 276 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 596 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 460 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 104
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 104 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 192
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 142 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 192 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 460 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00788] select: init of component sm returned success
[cn2:00788] select: initializing btl component tcp
[cn3:21245] found 1 verbs interface
[cn3:21245] examining verbs interface: mlx4_0
[cn3:21245] btl:usnic: no usNICs found
[cn3:21245] storing data in PEER dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 18166251521
[cn3:21245] select: init of component usnic returned failure
[cn3:21245] usNIC connectivity client finalized
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] message received 104 bytes from [[15051,0],1] for tag 2
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 192 bytes from [[15051,0],2] for tag 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 216 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 337
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 337 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00788] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn3:21245] mca: base: close: component usnic closed
[cn3:21245] mca: base: close: unloading component usnic
[cn3:21245] select: initializing btl component vader
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 276
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 192 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 276 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 79 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 337 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 337 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 276 bytes from [[15051,0],1] for tag 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 511 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 80 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 700
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 142
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 142 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 511
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 511 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21245] storing data in PEER dstore
[cn3:21245] dstore:hash:store storing data for proc 18166251521
[cn3:21245] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 18166251521
[cn3:21245] select: init of component vader returned success
[cn3:21245] select: init returned priority 20
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 66 BYTES ON SOCKET 9
[cn2:00788] btl: tcp: Found match: 127.0.0.1 (lo)
[cn2:00788] storing data in PEER dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 9576316929
[cn1:13986] [[15051,0],0] message received 142 bytes from [[15051,0],1] for tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 206 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 511 bytes from [[15051,0],2] for tag 2
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 76 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 79
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,1],9]
[cn5:13991] [[15051,0],4]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] RECVD COMPLETE MESSAGE FROM [[15051,1],9] (ORIGIN [[15051,1],9]) OF 700 BYTES FOR DEST [[15051,0],0] TAG 34
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 79 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4] ROUTING TO [[15051,0],0] FROM HERE
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:583] queue relay to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 162 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 206
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE RELAY COMPLETE TO [[15051,0],0] OF 700 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,1],9]
[cn5:13991] [[15051,0],4]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 206 BYTES FOR DEST [[15051,0],0] TAG 2
[cn5:13991] [[15051,0],4]-[[15051,1],9] mca_oob_tcp_msg_recv: peer closed connection
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4] tcp_peer_close for [[15051,1],9] sd 13 state CONNECTED
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] tcp:lost connection called for peer [[15051,1],9]
[cn2:00788] select: init of component tcp returned success
[cn2:00788] select: initializing btl component usnic
[cn5:13991] [[15051,0],4] route to [[15051,1],9] lost
[cn3:21245] selected ob1 best priority 20
[cn3:21245] select: component ob1 selected
[cn3:21245] mca: base: close: component v closed
[cn3:21245] mca: base: close: unloading component v
[cn3:21245] mca: base: close: component bfo closed
[cn3:21245] mca: base: close: unloading component bfo
[cn3:21245] [[15051,1],4] running modex
[cn3:21245] [[15051,1],4] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] message received 79 bytes from [[15051,0],1] for tag 2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 206 bytes from [[15051,0],2] for tag 2
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 112 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00788] usNIC connectivity agent initialized
[cn3:21246] [[15051,1],5] ORTE_ERROR_LOG: Connection reset by peer in file /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/usnic/btl_usnic_cclient.c at line 124
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,1],9]) OF 700 BYTES FOR DEST [[15051,0],0] TAG 34
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 80
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 80 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 104
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 112 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 700 bytes from [[15051,1],9] for tag 34
[cn1:13986] [[15051,0],0] got show_help from [[15051,1],9]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 328 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 80 bytes from [[15051,0],1] for tag 2
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00788] usNIC connectivity client initialized
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 104 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 66
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 66 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 66 bytes from [[15051,0],1] for tag 2
[cn2:00788] found 1 verbs interface
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 76
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 76 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 76 bytes from [[15051,0],1] for tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,1],5]
[cn3:21242] [[15051,0],2]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:tcp:recv:handler allocate data region of size 700
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00788] examining verbs interface: mlx4_0
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 162
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 162 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 104 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 162 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 104 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00789] usNIC connectivity client initialized
[cn2:00789] found 1 verbs interface
[cn2:00789] examining verbs interface: mlx4_0
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21246] [[15051,1],5] rml_send_buffer to peer [[15051,0],0] at tag 34
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1793 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 112
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 112 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1793
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1793 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 112 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0] message received 1793 bytes from [[15051,0],2] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 112
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 112 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00788] found 1 verbs interface
[cn2:00788] examining verbs interface: mlx4_0
[cn3:21246] [[15051,1],5] rml_send_msg to peer [[15051,0],0] at tag 34
[cn3:21246] [[15051,1],5] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21246] [[15051,1],5] oob:base:send to target [[15051,0],0]
[cn3:21246] [[15051,1],5] oob:base:send unknown peer [[15051,0],0]
[cn3:21246] fetching data from PEER dstore
[cn3:21246] dstore:hash:fetch: searching for key orte.rmluri on proc 986382336
[cn3:21246] dstore_hash:fetch key orte.rmluri for proc 986382336 not found
[cn3:21246] [[15051,1],5] routed_binomial_get([[15051,0],0]) --> [[15051,0],2]
[cn3:21246] [[15051,1],5] is reachable by TCP
[cn3:21246] [[15051,1],5] oob:tcp:send_nb to peer [[15051,0],0]:34
[cn3:21246] [[15051,1],5] tcp:send_nb to peer [[15051,0],0]
[cn3:21246] [[15051,1],5]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21246] [[15051,1],5]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:34
[cn3:21246] [[15051,1],5] routed_binomial_get([[15051,0],0]) --> [[15051,0],2]
[cn3:21246] [[15051,1],5] tcp:send_nb: already connected to [[15051,0],2] - queueing for send
[cn3:21246] [[15051,1],5]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21246] [[15051,1],5] tcp:send_handler called to send to peer [[15051,0],2]
[cn3:21246] [[15051,1],5] tcp:send_handler SENDING TO [[15051,0],2]
[cn3:21246] [[15051,1],5] MESSAGE SEND COMPLETE TO [[15051,0],2] OF 700 BYTES ON SOCKET 11
[cn3:21246] [[15051,1],5]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 112 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 328
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 328 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00789] found 1 verbs interface
[cn2:00789] examining verbs interface: mlx4_0
[cn1:13986] [[15051,0],0] message received 328 bytes from [[15051,0],1] for tag 2
[cn2:00788] btl:usnic: no usNICs found
[cn2:00788] storing data in PEER dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 9576316929
[cn2:00788] select: init of component usnic returned failure
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 93
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 93 BYTES FOR DEST [[15051,0],0] TAG 2
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 93 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 93 bytes from [[15051,0],2] for tag 2
[cn3:21246] sess_dir_finalize: proc session dir does not exist
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 700
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,1],5]
[cn3:21242] [[15051,0],2]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] RECVD COMPLETE MESSAGE FROM [[15051,1],5] (ORIGIN [[15051,1],5]) OF 700 BYTES FOR DEST [[15051,0],0] TAG 34
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] ROUTING TO [[15051,0],0] FROM HERE
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:583] queue relay to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,1],5]) OF 700 BYTES FOR DEST [[15051,0],0] TAG 34
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 700 bytes from [[15051,1],5] for tag 34
[cn1:13986] [[15051,0],0] got show_help from [[15051,1],5]
[cn3:21242] [[15051,0],2] MESSAGE RELAY COMPLETE TO [[15051,0],0] OF 700 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,1],5]
[cn3:21242] [[15051,0],2]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]-[[15051,1],5] mca_oob_tcp_msg_recv: peer closed connection
[cn3:21242] [[15051,0],2] tcp_peer_close for [[15051,1],5] sd 13 state CONNECTED
[cn3:21242] [[15051,0],2] tcp:lost connection called for peer [[15051,1],5]
[cn3:21242] [[15051,0],2] route to [[15051,1],5] lost
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 194
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 194 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 194 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 194 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4][[15051,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 219
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 219 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 219 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 219 bytes from [[15051,0],3] for tag 2
[cn4][[15051,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 79
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 79 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 79 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 79 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1188
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 1188 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17658] [rank=7] openib: using device mlx4_0
[cn1:13986] [[15051,0],0] message received 1188 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 462
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 462 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4][[15051,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn4][[15051,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn4][[15051,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn4][[15051,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn4][[15051,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn4][[15051,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn1:13986] [[15051,0],0] message received 462 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 166
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 166 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4][[15051,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn4][[15051,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x1936fc0 for btl 0x1942460
[cn1:13986] [[15051,0],0] message received 166 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17658] openib BTL: rdmacm IP address not found on port
[cn4:17658] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1188 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 462 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 166 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,1],2]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate data region of size 348
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,1],3]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate data region of size 348
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4126
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 4126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 4126 bytes from [[15051,0],1] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4126 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 4109
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 2
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00789] btl:usnic: no usNICs found
[cn2:00789] storing data in PEER dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 13871284225
[cn2:00789] select: init of component usnic returned failure
[cn2:00789] mca: base: close: component usnic closed
[cn2:00789] mca: base: close: unloading component usnic
[cn2:00789] select: initializing btl component vader
[cn2:00789] storing data in PEER dstore
[cn2:00789] dstore:hash:store storing data for proc 13871284225
[cn2:00789] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 13871284225
[cn2:00789] select: init of component vader returned success
[cn2:00789] select: init returned priority 20
[cn2:00789] selected ob1 best priority 20
[cn2:00789] select: component ob1 selected
[cn2:00789] mca: base: close: component v closed
[cn2:00789] mca: base: close: unloading component v
[cn2:00789] mca: base: close: component bfo closed
[cn2:00789] mca: base: close: unloading component bfo
[cn2:00789] [[15051,1],3] running modex
[cn2:00789] [[15051,1],3] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn2:00789] [[15051,1],3] grpcomm:base:modex: performing modex
[cn2:00789] fetching data from PEER dstore
[cn2:00789] dstore:hash:fetch: searching for key NULL on proc 13871284225
[cn2:00789] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 13871284225
[cn2:00789] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 13871284225
[cn2:00789] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 13871284225
[cn2:00789] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 13871284225
[cn2:00789] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 13871284225
[cn2:00789] [[15051,1],3] grpcomm:base:pack_modex: reporting 5 entries
[cn2:00789] [[15051,1],3] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn2:00789] [[15051,1],3] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn2:00789] [[15051,1],3] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn2:00789] [[15051,1],3] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn2:00789] [[15051,1],3] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn2:00789] [[15051,1],3] grpcomm:base:full:modex: executing allgather
[cn2:00789] [[15051,1],3] grpcomm:bad entering allgather
[cn2:00789] [[15051,1],3] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn2:00789] [[15051,1],3] grpcomm:base:modex: modex posted
[cn2:00789] [[15051,1],3] grpcomm:bad sending collective 0 to our daemon
[cn2:00789] [[15051,1],3] rml_send_buffer to peer [[15051,0],1] at tag 30
[cn2:00789] [[15051,1],3] grpcomm:bad allgather underway
[cn2:00789] [[15051,1],3] rml_send_msg to peer [[15051,0],1] at tag 30
[cn2:00789] [[15051,1],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00789] [[15051,1],3] oob:base:send to target [[15051,0],1]
[cn2:00789] [[15051,1],3] oob:base:send known transport for peer [[15051,0],1]
[cn2:00789] [[15051,1],3] oob:tcp:send_nb to peer [[15051,0],1]:30
[cn2:00789] [[15051,1],3] tcp:send_nb to peer [[15051,0],1]
[cn2:00789] [[15051,1],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],1]
[cn2:00789] [[15051,1],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],1]:30
[cn2:00789] [[15051,1],3] routed_binomial_get([[15051,0],1]) --> [[15051,0],1]
[cn2:00789] [[15051,1],3] tcp:send_nb: already connected to [[15051,0],1] - queueing for send
[cn2:00789] [[15051,1],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],1]
[cn2:00789] [[15051,1],3] tcp:send_handler called to send to peer [[15051,0],1]
[cn2:00789] [[15051,1],3] tcp:send_handler SENDING TO [[15051,0],1]
[cn[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 4109 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 273 BYTES ON SOCKET 9
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 4109 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 216
[cn1:13986] [[15051,0],0] message received 4109 bytes from [[15051,0],1] for tag 2
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 273
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 273 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 878
[cn2:00788] usNIC connectivity client finalized
[cn2:00788] mca: base: close: component usnic closed
[cn2:00788] mca: base: close: unloading component usnic
[cn2:00788] select: initializing btl component vader
[cn2:00788] storing data in PEER dstore
[cn2:00788] dstore:hash:store storing data for proc 9576316929
[cn2:00788] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 9576316929
[cn2:00788] select: init of component vader returned success
[cn2:00788] select: init returned priority 20
[cn2:00788] selected ob1 best priority 20
[cn2:00788] select: component ob1 selected
[cn2:00788] mca: base: close: component v closed
[cn2:00788] mca: base: close: unloading component v
[cn2:00788] mca: base: close: component bfo closed
[cn2:00788] mca: base: close: unloading component bfo
[cn2:00788] [[15051,1],2] running modex
[cn2:00788] [[15051,1],2] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn2:00788] [[15051,1],2] grpcomm:base:modex: performing modex
[cn2:00788] fetching data from PEER dstore
[cn2:00788] dstore:hash:fetch: searching for key NULL on proc 9576316929
[cn2:00788] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 9576316929
[cn2:00788] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 9576316929
[cn2:00788] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 9576316929
[cn2:00788] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 9576316929
[cn2:00788] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 9576316929
[cn2:00788] [[15051,1],2] grpcomm:base:pack_modex: reporting 5 entries
[cn2:00788] [[15051,1],2] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn2:00788] [[15051,1],2] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn2:00788] [[15051,1],2] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn2:00788] [[15051,1],2] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn2:00788] [[15051,1],2] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn2:00788] [[15051,1],2] grpcomm:base:full:modex: executing allgather
[cn2:00788] [[15051,1],2] grpcomm:bad entering allgather
[cn2:00788] [[15051,1],2] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn2:00788] [[15051,1],2] grpcomm:base:modex: modex posted
[cn2:00788] [[15051,1],2] grpcomm:bad sending collective 0 to our daemon
[cn2:00788] [[15051,1],2] rml_send_buffer to peer [[15051,0],1] at tag 30
[cn2:00788] [[15051,1],2] grpcomm:bad allgather underway
[cn2:00788] [[15051,1],2] rml_send_msg to peer [[15051,0],1] at tag 30
[cn2:00788] [[15051,1],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00788] [[15051,1],2] oob:base:send to target [[15051,0],1]
[cn2:00788] [[15051,1],2] oob:base:send known transport for peer [[15051,0],1]
[cn2:00788] [[15051,1],2] oob:tcp:send_nb to peer [[15051,0],1]:30
[cn2:00788] [[15051,1],2] tcp:send_nb to peer [[15051,0],1]
[cn2:00788] [[15051,1],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],1]
[cn2:00788] [[15051,1],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],1]:30
[cn2:00788] [[15051,1],2] routed_binomial_get([[15051,0],1]) --> [[15051,0],1]
[cn2:00788] [[15051,1],2] tcp:send_nb: already connected to [[15051,0],1] - queueing for send
[cn2:00788] [[15051,1],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],1]
[cn2:00788] [[15051,1],2] tcp:send_handler called to send to peer [[15051,0],1]
[cn2:00788] [[15051,1],2] tcp:send_handler SENDING TO [[15051,0],1]
[cn2:00788] [[15051,1],2] MESSAGE SEND COMPLETE TO [[15051,0],1] OF 348 BYTES ON SOCKET 11
[cn2:00788] [[15051,1],2]-[[15051,0],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 273 bytes from [[15051,0],1] for tag 2
2:00789] [[15051,1],3] MESSAGE SEND COMPLETE TO [[15051,0],1] OF 348 BYTES ON SOCKET 11
[cn2:00789] [[15051,1],3]-[[15051,0],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 878 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 878 bytes from [[15051,0],2] for tag 2
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 878 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21245] [[15051,1],4] grpcomm:base:modex: performing modex
[cn3:21245] fetching data from PEER dstore
[cn3:21245] dstore:hash:fetch: searching for key NULL on proc 18166251521
[cn3:21245] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 18166251521
[cn3:21245] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 18166251521
[cn3:21245] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 18166251521
[cn3:21245] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 18166251521
[cn3:21245] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 18166251521
[cn3:21245] [[15051,1],4] grpcomm:base:pack_modex: reporting 5 entries
[cn3:21245] [[15051,1],4] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn3:21245] [[15051,1],4] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 216 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1957
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1957 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 1957 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] message received 216 bytes from [[15051,0],3] for tag 2
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,1],4]
[cn1:13986] [[15051,0],0] message received 1957 bytes from [[15051,0],2] for tag 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 216 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:tcp:recv:handler allocate data region of size 348
[cn4][[15051,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn3:21245] [[15051,1],4] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn3:21245] [[15051,1],4] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn3:21245] [[15051,1],4] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn3:21245] [[15051,1],4] grpcomm:base:full:modex: executing allgather
[cn3:21245] [[15051,1],4] grpcomm:bad entering allgather
[cn3:21245] [[15051,1],4] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn3:21245] [[15051,1],4] grpcomm:base:modex: modex posted
[cn3:21245] [[15051,1],4] grpcomm:bad sending collective 0 to our daemon
[cn3:21245] [[15051,1],4] rml_send_buffer to peer [[15051,0],2] at tag 30
[cn3:21245] [[15051,1],4] grpcomm:bad allgather underway
[cn3:21245] [[15051,1],4] rml_send_msg to peer [[15051,0],2] at tag 30
[cn3:21245] [[15051,1],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21245] [[15051,1],4] oob:base:send to target [[15051,0],2]
[cn3:21245] [[15051,1],4] oob:base:send known transport for peer [[15051,0],2]
[cn3:21245] [[15051,1],4] oob:tcp:send_nb to peer [[15051,0],2]:30
[cn3:21245] [[15051,1],4] tcp:send_nb to peer [[15051,0],2]
[cn3:21245] [[15051,1],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],2]
[cn3:21245] [[15051,1],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],2]:30
[cn3:21245] [[15051,1],4] routed_binomial_get([[15051,0],2]) --> [[15051,0],2]
[cn3:21245] [[15051,1],4] tcp:send_nb: already connected to [[15051,0],2] - queueing for send
[cn3:21245] [[15051,1],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],2]
[cn3:21245] [[15051,1],4] tcp:send_handler called to send to peer [[15051,0],2]
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 2
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 344
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 344 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 344 bytes from [[15051,0],2] for tag 2
[cn3:21245] [[15051,1],4] tcp:send_handler SENDING TO [[15051,0],2]
[cn3:21245] [[15051,1],4] MESSAGE SEND COMPLETE TO [[15051,0],2] OF 348 BYTES ON SOCKET 11
[cn3:21245] [[15051,1],4]-[[15051,0],2] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 344 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 998
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 998 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] message received 998 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1047
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 1047 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17658] storing data in PEER dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 31051153409
[cn4:17658] select: init of component openib returned success
[cn4:17658] select: initializing btl component self
[cn4:17658] select: init of component self returned success
[cn4:17658] select: initializing btl component sm
[cn4:17658] select: init of component sm returned success
[cn4:17658] select: initializing btl component tcp
[cn4:17658] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn4:17658] btl: tcp: Found match: 127.0.0.1 (lo)
[cn4:17658] storing data in PEER dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 31051153409
[cn4:17658] select: init of component tcp returned success
[cn4:17658] select: initializing btl component usnic
[cn1:13986] [[15051,0],0] message received 1047 bytes from [[15051,0],3] for tag 2
[cn4:17657] storing data in PEER dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 26756186113
[cn4:17657] select: init of component openib returned success
[cn4:17657] select: initializing btl component self
[cn4:17657] select: init of component self returned success
[cn4:17657] select: initializing btl component sm
[cn4:17657] select: init of component sm returned success
[cn4:17657] select: initializing btl component tcp
[cn4:17657] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn4:17657] btl: tcp: Found match: 127.0.0.1 (lo)
[cn4:17657] storing data in PEER dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 26756186113
[cn4:17657] select: init of component tcp returned success
[cn4:17657] select: initializing btl component usnic
[cn4:17657] usNIC connectivity agent initialized
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 596
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 596 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 596 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 998 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1047 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 596 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 596
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 596 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4][[15051,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 5, qpn 32315
[cn4][[15051,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 5, Port: 1, QPN: 32315
[cn4][[15051,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn1:13986] [[15051,0],0] message received 596 bytes from [[15051,0],3] for tag 2
[cn4][[15051,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 5, qpn 32316
[cn4][[15051,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 5, Port: 1, QPN: 32316
[cn4][[15051,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 80
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 596 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 80 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 80 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17657] usNIC connectivity client initialized
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 80 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 112
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 112 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 112 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 112 bytes from [[15051,0],3] for tag 2
[cn4:17657] found 1 verbs interface
[cn4:17657] examining verbs interface: mlx4_0
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 80
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 80 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 80 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 80 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17658] usNIC connectivity client initialized
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 112
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 112 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 112 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17658] found 1 verbs interface
[cn4:17658] examining verbs interface: mlx4_0
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 112 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 112
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 112 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 112 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17657] found 1 verbs interface
[cn4:17657] examining verbs interface: mlx4_0
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 112 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 66
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 66 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 66 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] message received 66 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17658] found 1 verbs interface
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 76 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 76
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 76 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] message received 76 bytes from [[15051,0],3] for tag 2
[cn4:17658] examining verbs interface: mlx4_0
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 330
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 330 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 330 bytes from [[15051,0],3] for tag 2
[cn4:17657] btl:usnic: no usNICs found
[cn4:17657] storing data in PEER dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 26756186113
[cn4:17657] select: init of component usnic returned failure
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 187
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 187 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 187 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 83
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 83 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17657] usNIC connectivity client finalized
[cn4:17657] mca: base: close: component usnic closed
[cn4:17657] mca: base: close: unloading component usnic
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 330 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] message received 83 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17657] select: initializing btl component vader
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 291
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 187 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 83 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 291 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 291 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17657] storing data in PEER dstore
[cn4:17657] dstore:hash:store storing data for proc 26756186113
[cn4:17657] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 26756186113
[cn4:17657] select: init of component vader returned success
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 291 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 367
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 367 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn1:13986] [[15051,0],0] message received 367 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 367 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17657] select: init returned priority 20
[cn4:17657] selected ob1 best priority 20
[cn4:17657] select: component ob1 selected
[cn4:17657] mca: base: close: component v closed
[cn4:17657] mca: base: close: unloading component v
[cn4:17657] mca: base: close: component bfo closed
[cn4:17657] mca: base: close: unloading component bfo
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 220
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 220 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 220 bytes from [[15051,0],3] for tag 2
[cn4:17657] [[15051,1],6] running modex
[cn4:17657] [[15051,1],6] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 220 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 69
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 69 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 69 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 69 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17658] btl:usnic: no usNICs found
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 70
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 70 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 70 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 70 bytes from [[15051,0],3] for tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 94
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 94 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17658] storing data in PEER dstore
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 94 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 126
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 126 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 126 bytes from [[15051,0],3] for tag 2
[cn4:17658] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 31051153409
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 94 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:14001] [[15051,1],0]:tcp:recv:handler called for peer [[15051,0],0]
[cn1:14001] [[15051,1],0]:tcp:recv:handler CONNECTED
[cn1:14001] [[15051,1],0] RECVD COMPLETE MESSAGE FROM [[15051,0],0] (ORIGIN [[15051,0],0]) OF 14140 BYTES FOR DEST [[15051,1],0] TAG 20
[cn1:14001] [[15051,1],0] DELIVERING TO RML
[cn1:14001] [[15051,1],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14001] [[15051,1],0] message received 14140 bytes from [[15051,0],0] for tag 20
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:14001] [[15051,1],0] registering sync ack recvd
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1041
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 126 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 1041 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 1041 bytes from [[15051,0],3] for tag 2
[cn4:17658] select: init of component usnic returned failure
[cn4:17658] mca: base: close: component usnic closed
[cn4:17658] mca: base: close: unloading component usnic
[cn4:17658] select: initializing btl component vader
[cn4:17658] storing data in PEER dstore
[cn4:17658] dstore:hash:store storing data for proc 31051153409
[cn4:17658] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 31051153409
[cn4:17658] select: init of component vader returned success
[cn4:17658] select: init returned priority 20
[cn4:17658] selected ob1 best priority 20
[cn4:17658] select: component ob1 selected
[cn4:17658] mca: base: close: component v closed
[cn4:17658] mca: base: close: unloading component v
[cn4:17658] mca: base: close: component bfo closed
[cn4:17658] mca: base: close: unloading component bfo
[cn4:17658] [[15051,1],7] running modex
[cn4:17658] [[15051,1],7] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1041 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 3119
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,1],7]
[cn4:17654] [[15051,0],3]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]:tcp:recv:handler allocate data region of size 348
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 3119 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 3119 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 3119 bytes from [[15051,0],3] for tag 2
[cn4:17658] [[15051,1],7] grpcomm:base:modex: performing modex
[cn4:17658] fetching data from PEER dstore
[cn4:17658] dstore:hash:fetch: searching for key NULL on proc 31051153409
[cn4:17658] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 31051153409
[cn4:17658] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 31051153409
[cn4:17658] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 31051153409
[cn4:17658] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 31051153409
[cn4:17658] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 31051153409
[cn4:17658] [[15051,1],7] grpcomm:base:pack_modex: reporting 5 entries
[cn4:17658] [[15051,1],7] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn4:17658] [[15051,1],7] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn4:17658] [[15051,1],7] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn4:17658] [[15051,1],7] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn4:17658] [[15051,1],7] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn4:17658] [[15051,1],7] grpcomm:base:full:modex: executing allgather
[cn4:17658] [[15051,1],7] grpcomm:bad entering allgather
[cn4:17658] [[15051,1],7] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn4:17658] [[15051,1],7] grpcomm:base:modex: modex posted
[cn4:17658] [[15051,1],7] grpcomm:bad sending collective 0 to our daemon
[cn4:17658] [[15051,1],7] rml_send_buffer to peer [[15051,0],3] at tag 30
[cn4:17658] [[15051,1],7] grpcomm:bad allgather underway
[cn4:17658] [[15051,1],7] rml_send_msg to peer [[15051,0],3] at tag 30
[cn4:17658] [[15051,1],7] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17658] [[15051,1],7] oob:base:send to target [[15051,0],3]
[cn4:17658] [[15051,1],7] oob:base:send known transport for peer [[15051,0],3]
[cn4:17658] [[15051,1],7] oob:tcp:send_nb to peer [[15051,0],3]:30
[cn4:17658] [[15051,1],7] tcp:send_nb to peer [[15051,0],3]
[cn4:17658] [[15051,1],7]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],3]
[cn4:17658] [[15051,1],7]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],3]:30
[cn4:17658] [[15051,1],7] routed_binomial_get([[15051,0],3]) --> [[15051,0],3]
[cn4:17658] [[15051,1],7] tcp:send_nb: already connected to [[15051,0],3] - queueing for send
[cn4:17658] [[15051,1],7]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],3]
[cn4:17658] [[15051,1],7] tcp:send_handler called to send to peer [[15051,0],3]
[cn4:17658] [[15051,1],7] tcp:send_handler SENDING TO [[15051,0],3]
[cn4:17658] [[15051,1],7] MESSAGE SEND COMPLETE TO [[15051,0],3] OF 348 BYTES ON SOCKET 11
[cn4:17658] [[15051,1],7]-[[15051,0],3] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 986382336
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 986382336
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14001] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 986382337
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 986382337
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 986382337
[cn1:14001] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 986382337
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 5281349632
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349632
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14001] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 9576316928
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316928
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14001] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 13871284224
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284224
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14001] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 18166251520
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251520
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14001] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 18446744070400966657
[cn1:14001] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 18446744070400966657
[cn1:14001] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070400966657
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 986382337
[cn1:14001] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 986382337
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 986382337
[cn1:14001] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 986382337
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 986382337
[cn1:14001] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 986382337
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 986382337
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 986382337
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 986382337
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 986382337
[cn1:14001] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 986382337
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 5281349633
[cn1:14001] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5281349633
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 5281349633
[cn1:14001] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5281349633
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 5281349633
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 5281349633
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5281349633
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 5281349633
[cn1:14001] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5281349633
[cn1:14001] fetching data from INTERNAL dstore
[cn1:14001] dstore:hash:fetch: searching for key orte.hostname on proc 986382336
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 5281349633
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5281349633
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 5281349633
[cn1:14001] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5281349633
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 9576316929
[cn1:14001] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9576316929
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 9576316929
[cn1:14001] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9576316929
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 9576316929
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 9576316929
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9576316929
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 9576316929
[cn1:14001] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9576316929
[cn1:14001] fetching data from INTERNAL dstore
[cn1:14001] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 9576316929
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9576316929
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 9576316929
[cn1:14001] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9576316929
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 13871284225
[cn1:14001] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13871284225
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 13871284225
[cn1:14001] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13871284225
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 13871284225
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 13871284225
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13871284225
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 13871284225
[cn1:14001] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13871284225
[cn1:14001] fetching data from INTERNAL dstore
[cn1:14001] dstore:hash:fetch: searching for key orte.hostname on proc 5281349632
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 13871284225
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13871284225
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 13871284225
[cn1:14001] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13871284225
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 18166251521
[cn1:14001] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18166251521
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 18166251521
[cn1:14001] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18166251521
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 18166251521
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 18166251521
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18166251521
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 18166251521
[cn1:14001] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18166251521
[cn1:14001] fetching data from INTERNAL dstore
[cn1:14001] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 18166251521
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18166251521
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 18166251521
[cn1:14001] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18166251521
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 22461218817
[cn1:14001] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22461218817
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 22461218817
[cn1:14001] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22461218817
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 22461218817
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 22461218817
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22461218817
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 22461218817
[cn1:14001] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22461218817
[cn1:14001] fetching data from INTERNAL dstore
[cn1:14001] dstore:hash:fetch: searching for key orte.hostname on proc 9576316928
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 22461218817
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22461218817
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 22461218817
[cn1:14001] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22461218817
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 26756186113
[cn1:14001] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26756186113
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 26756186113
[cn1:14001] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26756186113
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 26756186113
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 26756186113
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26756186113
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 26756186113
[cn1:14001] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26756186113
[cn1:14001] fetching data from INTERNAL dstore
[cn1:14001] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 26756186113
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26756186113
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 26756186113
[cn1:14001] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26756186113
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 31051153409
[cn1:14001] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31051153409
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 31051153409
[cn1:14001] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31051153409
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 31051153409
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 31051153409
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31051153409
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 31051153409
[cn1:14001] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31051153409
[cn1:14001] fetching data from INTERNAL dstore
[cn1:14001] dstore:hash:fetch: searching for key orte.hostname on proc 13871284224
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 31051153409
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31051153409
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 31051153409
[cn1:14001] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31051153409
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 35346120705
[cn1:14001] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35346120705
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 35346120705
[cn1:14001] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35346120705
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 35346120705
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 35346120705
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35346120705
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 35346120705
[cn1:14001] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35346120705
[cn1:14001] fetching data from INTERNAL dstore
[cn1:14001] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 35346120705
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35346120705
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 35346120705
[cn1:14001] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35346120705
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 39641088001
[cn1:14001] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39641088001
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 39641088001
[cn1:14001] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39641088001
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 39641088001
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 39641088001
[cn1:14001] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39641088001
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 39641088001
[cn1:14001] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39641088001
[cn1:14001] fetching data from INTERNAL dstore
[cn1:14001] dstore:hash:fetch: searching for key orte.hostname on proc 18166251520
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 39641088001
[cn1:14001] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39641088001
[cn1:14001] storing data in INTERNAL dstore
[cn1:14001] dstore:hash:store storing data for proc 39641088001
[cn1:14001] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39641088001
[cn1:14001] [[15051,1],0] Process bound at launch
[cn1:14001] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B/././.][./././.]
[cn1:14001] storing data in NONPEER dstore
[cn1:14001] dstore:hash:store storing data for proc 986382337
[cn1:14001] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 986382337
[cn1:14001] storing data in PEER dstore
[cn1:14001] dstore:hash:store storing data for proc 986382337
[cn1:14001] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 986382337
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 93
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 93 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 93 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 93 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1399
[cn4:17657] [[15051,1],6] grpcomm:base:modex: performing modex
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1399 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,1],6]
[cn4:17654] [[15051,0],3]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn4:17654] [[15051,0],3]:tcp:recv:handler allocate data region of size 348
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 2
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 1399 BYTES FOR DEST [[15051,0],0] TAG 2
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1441 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] message received 1399 bytes from [[15051,0],3] for tag 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 2
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17657] fetching data from PEER dstore
[cn4:17657] dstore:hash:fetch: searching for key NULL on proc 26756186113
[cn4:17657] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 26756186113
[cn4:17657] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 26756186113
[cn4:17657] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 26756186113
[cn4:17657] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 26756186113
[cn4:17657] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 26756186113
[cn4:17657] [[15051,1],6] grpcomm:base:pack_modex: reporting 5 entries
[cn4:17657] [[15051,1],6] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn4:17657] [[15051,1],6] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn4:17657] [[15051,1],6] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn4:17657] [[15051,1],6] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn4:17657] [[15051,1],6] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn4:17657] [[15051,1],6] grpcomm:base:full:modex: executing allgather
[cn4:17657] [[15051,1],6] grpcomm:bad entering allgather
[cn4:17657] [[15051,1],6] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn4:17657] [[15051,1],6] grpcomm:base:modex: modex posted
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1441
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 1441 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 1441 bytes from [[15051,0],3] for tag 2
[cn4:17657] [[15051,1],6] grpcomm:bad sending collective 0 to our daemon
[cn4:17657] [[15051,1],6] rml_send_buffer to peer [[15051,0],3] at tag 30
[cn4:17657] [[15051,1],6] grpcomm:bad allgather underway
[cn4:17657] [[15051,1],6] rml_send_msg to peer [[15051,0],3] at tag 30
[cn4:17657] [[15051,1],6] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17657] [[15051,1],6] oob:base:send to target [[15051,0],3]
[cn4:17657] [[15051,1],6] oob:base:send known transport for peer [[15051,0],3]
[cn4:17657] [[15051,1],6] oob:tcp:send_nb to peer [[15051,0],3]:30
[cn4:17657] [[15051,1],6] tcp:send_nb to peer [[15051,0],3]
[cn4:17657] [[15051,1],6]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],3]
[cn4:17657] [[15051,1],6]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],3]:30
[cn4:17657] [[15051,1],6] routed_binomial_get([[15051,0],3]) --> [[15051,0],3]
[cn4:17657] [[15051,1],6] tcp:send_nb: already connected to [[15051,0],3] - queueing for send
[cn4:17657] [[15051,1],6]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],3]
[cn4:17657] [[15051,1],6] tcp:send_handler called to send to peer [[15051,0],3]
[cn4:17657] [[15051,1],6] tcp:send_handler SENDING TO [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 276
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 276 BYTES FOR DEST [[15051,0],0] TAG 2
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 276 bytes from [[15051,0],3] for tag 2
[cn4:17657] [[15051,1],6] MESSAGE SEND COMPLETE TO [[15051,0],3] OF 348 BYTES ON SOCKET 11
[cn4:17657] [[15051,1],6]-[[15051,0],3] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:2
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 276 BYTES ON SOCKET 9
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14001] mca: base: components_register: registering bml components
[cn1:14001] mca: base: components_register: found loaded component r2
[cn1:14001] mca: base: components_register: component r2 register function successful
[cn1:14001] mca: base: components_open: opening bml components
[cn1:14001] mca: base: components_open: found loaded component r2
[cn1:14001] mca: base: components_open: component r2 open function successful
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 62
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 5
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 5
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:5
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:5
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 62 BYTES FOR DEST [[15051,0],0] TAG 5
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 62 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] message received 62 bytes from [[15051,0],2] for tag 5
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] plm:base:receive processing msg
[cn1:13986] [[15051,0],0] plm:base:receive update proc state command from [[15051,0],2]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for job [15051,1]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 5 state EXITED WITH NON-ZERO STATUS exit_code 1
[cn1:13986] [[15051,0],0] plm:base:receive done processing commands
[cn1:13986] [[15051,0],0]:/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/errmgr/default_hnp/errmgr_default_hnp.c(504) updating exit status to 1
[cn1:13986] [[15051,0],0] plm:base:orted_cmd sending orted_exit commands
[cn1:13986] [[15051,0],0] grpcomm:bad:xcast sent to job [15051,0] tag 1
[cn1:13986] [[15051,0],0] rml_send_buffer to peer [[15051,0],0] at tag 15
[cn1:13986] [[15051,0],0] rml_send_msg to peer [[15051,0],0] at tag 15
[cn1:13986] [[15051,0],0] rml_send_iovec_to_self at tag 15
[cn1:13986] [[15051,0],0] rml_send_to_self callback executing for tag 15
[cn1:13986] [[15051,0],0] message received 8 bytes from [[15051,0],0] for tag 15
[cn1:13986] [[15051,0],0] grpcomm:xcast:recv: with 8 bytes
[cn1:13986] [[15051,0],0] orte:daemon:send_relay sending relay msg to [[15051,0],1]
[cn1:13986] [[15051,0],0] rml_send_buffer to peer [[15051,0],1] at tag 15
[cn1:13986] [[15051,0],0] orte:daemon:send_relay sending relay msg to [[15051,0],2]
[cn1:13986] [[15051,0],0] rml_send_buffer to peer [[15051,0],2] at tag 15
[cn1:13986] [[15051,0],0] orte:daemon:send_relay sending relay msg to [[15051,0],4]
[cn1:13986] [[15051,0],0] rml_send_buffer to peer [[15051,0],4] at tag 15
[cn1:13986] [[15051,0],0] rml_send_buffer to peer [[15051,0],0] at tag 1
[cn1:13986] [[15051,0],0] rml_send_msg to peer [[15051,0],1] at tag 15
[cn1:13986] [[15051,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] rml_send_msg to peer [[15051,0],2] at tag 15
[cn1:13986] [[15051,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] rml_send_msg to peer [[15051,0],4] at tag 15
[cn1:13986] [[15051,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0] rml_send_msg to peer [[15051,0],0] at tag 1
[cn1:13986] [[15051,0],0] rml_send_iovec_to_self at tag 1
[cn1:13986] [[15051,0],0] oob:base:send to target [[15051,0],1]
[cn1:13986] [[15051,0],0] oob:base:send known transport for peer [[15051,0],1]
[cn1:13986] [[15051,0],0] oob:tcp:send_nb to peer [[15051,0],1]:15
[cn1:13986] [[15051,0],0] tcp:send_nb to peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],1]
[cn1:13986] [[15051,0],0] oob:base:send to target [[15051,0],2]
[cn1:13986] [[15051,0],0] oob:base:send known transport for peer [[15051,0],2]
[cn1:13986] [[15051,0],0] oob:tcp:send_nb to peer [[15051,0],2]:15
[cn1:13986] [[15051,0],0] tcp:send_nb to peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],2]
[cn1:13986] [[15051,0],0] oob:base:send to target [[15051,0],4]
[cn1:13986] [[15051,0],0] oob:base:send known transport for peer [[15051,0],4]
[cn1:13986] [[15051,0],0] oob:tcp:send_nb to peer [[15051,0],4]:15
[cn1:13986] [[15051,0],0] tcp:send_nb to peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],4]
[cn1:13986] [[15051,0],0] rml_send_to_self callback executing for tag 1
[cn1:13986] [[15051,0],0] message received 8 bytes from [[15051,0],0] for tag 1
[cn1:13986] [[15051,0],0] orted:comm:process_commands() Processing Command: ORTE_DAEMON_EXIT_CMD
[cn1:13986] [[15051,0],0] orted_cmd: received exit cmd
[cn1:13986] sess_dir_finalize: proc session dir does not exist
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 5
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 5
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:5
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:5
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 62 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] sess_dir_finalize: proc session dir does not exist
[cn1:13986] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn1:13986] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn1:13986] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn1:13986] [[15051,0],0] num routes 3
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],1]:15
[cn1:13986] [[15051,0],0] routed_binomial_get([[15051,0],1]) --> [[15051,0],1]
[cn1:13986] [[15051,0],0] tcp:send_nb: already connected to [[15051,0],1] - queueing for send
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],1]
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],2]:15
[cn1:13986] [[15051,0],0] routed_binomial_get([[15051,0],2]) --> [[15051,0],2]
[cn1:13986] [[15051,0],0] tcp:send_nb: already connected to [[15051,0],2] - queueing for send
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],2]
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],4]:15
[cn1:13986] [[15051,0],0] routed_binomial_get([[15051,0],4]) --> [[15051,0],4]
[cn1:13986] [[15051,0],0] tcp:send_nb: already connected to [[15051,0],4] - queueing for send
[cn1:13986] [[15051,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 62
[cn1:13986] [[15051,0],0] tcp:send_handler called to send to peer [[15051,0],4]
[cn1:13986] [[15051,0],0] tcp:send_handler SENDING TO [[15051,0],4]
[cn1:13986] [[15051,0],0] MESSAGE SEND COMPLETE TO [[15051,0],4] OF 8 BYTES ON SOCKET 14
[cn1:13986] [[15051,0],0]-[[15051,0],4] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] tcp:send_handler called to send to peer [[15051,0],1]
[cn1:13986] [[15051,0],0] tcp:send_handler SENDING TO [[15051,0],1]
[cn1:13986] [[15051,0],0] MESSAGE SEND COMPLETE TO [[15051,0],1] OF 8 BYTES ON SOCKET 15
[cn1:13986] [[15051,0],0]-[[15051,0],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] tcp:send_handler called to send to peer [[15051,0],2]
[cn1:13986] [[15051,0],0] tcp:send_handler SENDING TO [[15051,0],2]
[cn1:13986] [[15051,0],0] MESSAGE SEND COMPLETE TO [[15051,0],2] OF 8 BYTES ON SOCKET 17
[cn1:13986] [[15051,0],0]-[[15051,0],2] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,1],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]-[[15051,1],0] mca_oob_tcp_msg_recv: peer closed connection
[cn1:13986] [[15051,0],0] tcp_peer_close for [[15051,1],0] sd 20 state CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,1],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]-[[15051,1],1] mca_oob_tcp_msg_recv: peer closed connection
[cn1:13986] [[15051,0],0] tcp_peer_close for [[15051,1],1] sd 23 state CONNECTED
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate data region of size 8
[cn1:13986] [[15051,0],0] tcp:lost connection called for peer [[15051,1],0]
[cn1:13986] [[15051,0],0] route to [[15051,1],0] lost
[cn1:13986] [[15051,0],0] tcp:lost connection called for peer [[15051,1],1]
[cn1:13986] [[15051,0],0] route to [[15051,1],1] lost
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 62 BYTES FOR DEST [[15051,0],0] TAG 5
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn5:13991] [[15051,0],4]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]:tcp:recv:handler allocate data region of size 8
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 62 bytes from [[15051,0],4] for tag 5
[cn1:13986] [[15051,0],0] plm:base:receive processing msg
[cn1:13986] [[15051,0],0] plm:base:receive update proc state command from [[15051,0],4]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for job [15051,1]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 9 state EXITED WITH NON-ZERO STATUS exit_code 1
[cn1:13986] [[15051,0],0] plm:base:receive done processing commands
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]:tcp:recv:handler allocate data region of size 8
[cn1:13986] sess_dir_finalize: proc session dir does not exist
[cn1:13986] sess_dir_finalize: proc session dir does not exist
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,1],8]
[cn5:13991] [[15051,0],4]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] RECVD COMPLETE MESSAGE FROM [[15051,1],8] (ORIGIN [[15051,1],8]) OF 348 BYTES FOR DEST [[15051,0],4] TAG 30
[cn5:13991] [[15051,0],4] DELIVERING TO RML
[cn5:13991] [[15051,0],4] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] message received 348 bytes from [[15051,1],8] for tag 30
[cn5:13991] [[15051,0],4] COLLECTIVE RECVD FROM [[15051,1],8]
[cn5:13991] [[15051,0],4] WORKING COLLECTIVE 0
[cn5:13991] [[15051,0],4] PROGRESSING COLLECTIVE 0
[cn5:13991] [[15051,0],4] PROGRESSING COLL id 0
[cn5:13991] [[15051,0],4] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 1
[cn5:13991] [[15051,0],4] COLLECTIVE 0 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],4] at tag 32
[cn5:13991] [[15051,0],4] PROGRESSING COLL id 1
[cn5:13991] [[15051,0],4] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 1
[cn5:13991] [[15051,0],4] PROGRESSING COLL id 2
[cn5:13991] [[15051,0],4] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 1
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],4] at tag 32
[cn5:13991] [[15051,0],4] rml_send_iovec_to_self at tag 32
[cn5:13991] [[15051,0],4] rml_send_to_self callback executing for tag 32
[cn5:13991] [[15051,0],4] message received 370 bytes from [[15051,0],4] for tag 32
[cn5:13991] [[15051,0],4] grpcomm:base:daemon_coll: daemon collective recvd from [[15051,0],4]
[cn5:13991] [[15051,0],4] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn5:13991] [[15051,0],4] grpcomm:base:daemon_coll: NUM CONTRIBS: 1
[cn5:13991] [[15051,0],4] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[15051,0],0]
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 32
[cn5:13991] [[15051,0],4] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 10 ngr 1
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 32
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:32
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:32
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 370 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,1],3]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] RECVD COMPLETE MESSAGE FROM [[15051,1],3] (ORIGIN [[15051,1],3]) OF 348 BYTES FOR DEST [[15051,0],1] TAG 30
[cn2:00785] [[15051,0],1] DELIVERING TO RML
[cn2:00785] [[15051,0],1] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] message received 348 bytes from [[15051,1],3] for tag 30
[cn2:00785] [[15051,0],1] COLLECTIVE RECVD FROM [[15051,1],3]
[cn2:00785] [[15051,0],1] WORKING COLLECTIVE 0
[cn2:00785] [[15051,0],1] PROGRESSING COLLECTIVE 0
[cn2:00785] [[15051,0],1] PROGRESSING COLL id 0
[cn2:00785] [[15051,0],1] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn2:00785] [[15051,0],1] PROGRESSING COLL id 1
[cn2:00785] [[15051,0],1] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn2:00785] [[15051,0],1] PROGRESSING COLL id 2
[cn2:00785] [[15051,0],1] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,1],2]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] RECVD COMPLETE MESSAGE FROM [[15051,1],2] (ORIGIN [[15051,1],2]) OF 348 BYTES FOR DEST [[15051,0],1] TAG 30
[cn2:00785] [[15051,0],1] DELIVERING TO RML
[cn2:00785] [[15051,0],1] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] message received 348 bytes from [[15051,1],2] for tag 30
[cn2:00785] [[15051,0],1] COLLECTIVE RECVD FROM [[15051,1],2]
[cn2:00785] [[15051,0],1] WORKING COLLECTIVE 0
[cn2:00785] [[15051,0],1] PROGRESSING COLLECTIVE 0
[cn2:00785] [[15051,0],1] PROGRESSING COLL id 0
[cn2:00785] [[15051,0],1] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn2:00785] [[15051,0],1] COLLECTIVE 0 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],1] at tag 32
[cn2:00785] [[15051,0],1] PROGRESSING COLL id 1
[cn2:00785] [[15051,0],1] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn2:00785] [[15051,0],1] PROGRESSING COLL id 2
[cn2:00785] [[15051,0],1] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],1] at tag 32
[cn2:00785] [[15051,0],1] rml_send_iovec_to_self at tag 32
[cn2:00785] [[15051,0],1] rml_send_to_self callback executing for tag 32
[cn2:00785] [[15051,0],1] message received 708 bytes from [[15051,0],1] for tag 32
[cn2:00785] [[15051,0],1] grpcomm:base:daemon_coll: daemon collective recvd from [[15051,0],1]
[cn2:00785] [[15051,0],1] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn2:00785] [[15051,0],1] grpcomm:base:daemon_coll: NUM CONTRIBS: 2
[cn2:00785] [[15051,0],1] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 10 ngr 2
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,1],4]
[cn3:21242] [[15051,0],2]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] RECVD COMPLETE MESSAGE FROM [[15051,1],4] (ORIGIN [[15051,1],4]) OF 348 BYTES FOR DEST [[15051,0],2] TAG 30
[cn3:21242] [[15051,0],2] DELIVERING TO RML
[cn3:21242] [[15051,0],2] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] message received 348 bytes from [[15051,1],4] for tag 30
[cn3:21242] [[15051,0],2] COLLECTIVE RECVD FROM [[15051,1],4]
[cn3:21242] [[15051,0],2] WORKING COLLECTIVE 0
[cn3:21242] [[15051,0],2] PROGRESSING COLLECTIVE 0
[cn3:21242] [[15051,0],2] PROGRESSING COLL id 0
[cn3:21242] [[15051,0],2] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 1
[cn3:21242] [[15051,0],2] COLLECTIVE 0 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],2] at tag 32
[cn3:21242] [[15051,0],2] PROGRESSING COLL id 1
[cn3:21242] [[15051,0],2] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 1
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] PROGRESSING COLL id 2
[cn3:21242] [[15051,0],2] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 1
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],2] at tag 32
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] rml_send_iovec_to_self at tag 32
[cn3:21242] [[15051,0],2] rml_send_to_self callback executing for tag 32
[cn3:21242] [[15051,0],2] message received 370 bytes from [[15051,0],2] for tag 32
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2] grpcomm:base:daemon_coll: daemon collective recvd from [[15051,0],2]
[cn3:21242] [[15051,0],2] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2] grpcomm:base:daemon_coll: NUM CONTRIBS: 1
[cn3:21242] [[15051,0],2] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[15051,0],0]
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 32
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 370
[cn3:21242] [[15051,0],2] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 10 ngr 1
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 32
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:32
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:32
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 370 BYTES FOR DEST [[15051,0],0] TAG 32
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 370 bytes from [[15051,0],2] for tag 32
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 370 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[15051,0],2]
[cn1:13986] [[15051,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn1:13986] [[15051,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 1
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] grpcomm:base:daemon_coll: CANNOT PROGRESS
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2] RECVD COMPLETE MESSAGE FROM [[15051,0],0] (ORIGIN [[15051,0],0]) OF 8 BYTES FOR DEST [[15051,0],2] TAG 15
[cn3:21242] [[15051,0],2] DELIVERING TO RML
[cn3:21242] [[15051,0],2] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21242] [[15051,0],2] message received 8 bytes from [[15051,0],0] for tag 15
[cn3:21242] [[15051,0],2] grpcomm:xcast:recv: with 8 bytes
[cn3:21242] [[15051,0],2] orte:daemon:send_relay - recipient list is empty!
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],2] at tag 1
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],2] at tag 1
[cn3:21242] [[15051,0],2] rml_send_iovec_to_self at tag 1
[cn3:21242] [[15051,0],2] rml_send_to_self callback executing for tag 1
[cn3:21242] [[15051,0],2] message received 8 bytes from [[15051,0],2] for tag 1
[cn3:21242] [[15051,0],2] orted:comm:process_commands() Processing Command: ORTE_DAEMON_EXIT_CMD
[cn3:21242] [[15051,0],2] orted_cmd: received exit cmd
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,1],7]
[cn4:17654] [[15051,0],3]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] RECVD COMPLETE MESSAGE FROM [[15051,1],7] (ORIGIN [[15051,1],7]) OF 348 BYTES FOR DEST [[15051,0],3] TAG 30
[cn4:17654] [[15051,0],3] DELIVERING TO RML
[cn4:17654] [[15051,0],3] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] message received 348 bytes from [[15051,1],7] for tag 30
[cn4:17654] [[15051,0],3] COLLECTIVE RECVD FROM [[15051,1],7]
[cn4:17654] [[15051,0],3] WORKING COLLECTIVE 0
[cn4:17654] [[15051,0],3] PROGRESSING COLLECTIVE 0
[cn4:17654] [[15051,0],3] PROGRESSING COLL id 0
[cn4:17654] [[15051,0],3] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn4:17654] [[15051,0],3] PROGRESSING COLL id 1
[cn4:17654] [[15051,0],3] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn4:17654] [[15051,0],3] PROGRESSING COLL id 2
[cn4:17654] [[15051,0],3] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,1],6]
[cn4:17654] [[15051,0],3]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] RECVD COMPLETE MESSAGE FROM [[15051,1],6] (ORIGIN [[15051,1],6]) OF 348 BYTES FOR DEST [[15051,0],3] TAG 30
[cn4:17654] [[15051,0],3] DELIVERING TO RML
[cn4:17654] [[15051,0],3] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3] message received 348 bytes from [[15051,1],6] for tag 30
[cn4:17654] [[15051,0],3] COLLECTIVE RECVD FROM [[15051,1],6]
[cn4:17654] [[15051,0],3] WORKING COLLECTIVE 0
[cn4:17654] [[15051,0],3] PROGRESSING COLLECTIVE 0
[cn4:17654] [[15051,0],3] PROGRESSING COLL id 0
[cn4:17654] [[15051,0],3] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn4:17654] [[15051,0],3] COLLECTIVE 0 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],3] at tag 32
[cn4:17654] [[15051,0],3] PROGRESSING COLL id 1
[cn4:17654] [[15051,0],3] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn4:17654] [[15051,0],3] PROGRESSING COLL id 2
[cn4:17654] [[15051,0],3] ALL LOCAL PROCS FOR JOB [15051,1] CONTRIBUTE 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],3] at tag 32
[cn4:17654] [[15051,0],3] rml_send_iovec_to_self at tag 32
[cn4:17654] [[15051,0],3] rml_send_to_self callback executing for tag 32
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,0],3]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate data region of size 708
[cn4:17654] [[15051,0],3] message received 708 bytes from [[15051,0],3] for tag 32
[cn4:17654] [[15051,0],3] grpcomm:base:daemon_coll: daemon collective recvd from [[15051,0],3]
[cn4:17654] [[15051,0],3] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn4:17654] [[15051,0],3] grpcomm:base:daemon_coll: NUM CONTRIBS: 2
[cn4:17654] [[15051,0],3] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[15051,0],1]
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],1] at tag 32
[cn4:17654] [[15051,0],3] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 10 ngr 2
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],1] at tag 32
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],1]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],1]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],1]:32
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],1]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],1]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],1]:32
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],1]) --> [[15051,0],1]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],1] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],1]
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],1]
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],1]
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],1] OF 708 BYTES ON SOCKET 10
[cn4:17654] [[15051,0],3]-[[15051,0],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] RECVD COMPLETE MESSAGE FROM [[15051,0],0] (ORIGIN [[15051,0],0]) OF 8 BYTES FOR DEST [[15051,0],1] TAG 15
[cn2:00785] [[15051,0],1] DELIVERING TO RML
[cn2:00785] [[15051,0],1] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] message received 8 bytes from [[15051,0],0] for tag 15
[cn2:00785] [[15051,0],1] grpcomm:xcast:recv: with 8 bytes
[cn2:00785] [[15051,0],1] orte:daemon:send_relay sending relay msg to [[15051,0],3]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],3] at tag 15
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],1] at tag 1
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],3] at tag 15
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],1] at tag 1
[cn2:00785] [[15051,0],1] rml_send_iovec_to_self at tag 1
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],3]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],3]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],3]:15
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],3]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],3]
[cn2:00785] [[15051,0],1] rml_send_to_self callback executing for tag 1
[cn2:00785] [[15051,0],1] message received 8 bytes from [[15051,0],1] for tag 1
[cn2:00785] [[15051,0],1] orted:comm:process_commands() Processing Command: ORTE_DAEMON_EXIT_CMD
[cn2:00785] [[15051,0],1] orted_cmd: received exit cmd
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 370
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 370 BYTES FOR DEST [[15051,0],0] TAG 32
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 370 bytes from [[15051,0],4] for tag 32
[cn1:13986] [[15051,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[15051,0],4]
[cn1:13986] [[15051,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn1:13986] [[15051,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 1
[cn1:13986] [[15051,0],0] grpcomm:base:daemon_coll: CANNOT PROGRESS
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4] RECVD COMPLETE MESSAGE FROM [[15051,0],0] (ORIGIN [[15051,0],0]) OF 8 BYTES FOR DEST [[15051,0],4] TAG 15
[cn5:13991] [[15051,0],4] DELIVERING TO RML
[cn5:13991] [[15051,0],4] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:13991] [[15051,0],4] message received 8 bytes from [[15051,0],0] for tag 15
[cn5:13991] [[15051,0],4] grpcomm:xcast:recv: with 8 bytes
[cn5:13991] [[15051,0],4] orte:daemon:send_relay - recipient list is empty!
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],4] at tag 1
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],4] at tag 1
[cn5:13991] [[15051,0],4] rml_send_iovec_to_self at tag 1
[cn5:13991] [[15051,0],4] rml_send_to_self callback executing for tag 1
[cn5:13991] [[15051,0],4] message received 8 bytes from [[15051,0],4] for tag 1
[cn5:13991] [[15051,0],4] orted:comm:process_commands() Processing Command: ORTE_DAEMON_EXIT_CMD
[cn5:13991] [[15051,0],4] orted_cmd: received exit cmd
[cn3:21242] sess_dir_finalize: proc session dir does not exist
[cn3:21242] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn3:21242] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn3:21242] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn3:21242] [[15051,0],2] num routes 0
[cn3:21242] [[15051,0],2]:tcp:recv:handler called for peer [[15051,1],4]
[cn3:21242] [[15051,0],2]:tcp:recv:handler CONNECTED
[cn3:21242] [[15051,0],2]:tcp:recv:handler allocate new recv msg
[cn3:21242] [[15051,0],2]:tcp:recv:handler read hdr
[cn3:21242] [[15051,0],2]-[[15051,1],4] mca_oob_tcp_msg_recv: peer closed connection
[cn3:21242] [[15051,0],2] tcp_peer_close for [[15051,1],4] sd 12 state CONNECTED
[cn3:21242] [[15051,0],2] tcp:lost connection called for peer [[15051,1],4]
[cn3:21242] [[15051,0],2] route to [[15051,1],4] lost
[cn3:21242] sess_dir_finalize: proc session dir does not exist
[cn3:21242] [[15051,0],2] rml_send_buffer to peer [[15051,0],0] at tag 5
[cn3:21242] [[15051,0],2] num routes 0
[cn3:21242] [[15051,0],2] rml_send_msg to peer [[15051,0],0] at tag 5
[cn3:21242] [[15051,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21242] [[15051,0],2] oob:base:send to target [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:base:send known transport for peer [[15051,0],0]
[cn3:21242] [[15051,0],2] oob:tcp:send_nb to peer [[15051,0],0]:5
[cn3:21242] [[15051,0],2] tcp:send_nb to peer [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:5
[cn3:21242] [[15051,0],2] routing direct to the HNP
[cn3:21242] [[15051,0],2] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn3:21242] [[15051,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler called to send to peer [[15051,0],0]
[cn3:21242] [[15051,0],2] tcp:send_handler SENDING TO [[15051,0],0]
[cn3:21242] [[15051,0],2] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 73 BYTES ON SOCKET 9
[cn3:21242] [[15051,0],2]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21242] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn3:21242] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn3:21242] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn3:21242] [[15051,0],2] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 21
[cn3:21242] [[15051,0],2] grpcomm:base:receive stop comm
[cn3:21242] [[15051,0],2] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 30
[cn3:21242] [[15051,0],2] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 15
[cn3:21242] [[15051,0],2] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 32
[cn3:21242] mca: base: close: component bad closed
[cn3:21242] mca: base: close: unloading component bad
[cn3:21242] [[15051,0],2] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 3
[cn3:21242] [[15051,0],2] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 45
[cn3:21242] [[15051,0],2] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 46
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 73
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],2] (ORIGIN [[15051,0],2]) OF 73 BYTES FOR DEST [[15051,0],0] TAG 5
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 73 bytes from [[15051,0],2] for tag 5
[cn1:13986] [[15051,0],0] plm:base:receive processing msg
[cn1:13986] [[15051,0],0] plm:base:receive update proc state command from [[15051,0],2]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for job [15051,1]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 4 state NORMALLY TERMINATED exit_code 11
[cn1:13986] [[15051,0],0] plm:base:receive done processing commands
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]-[[15051,0],2] mca_oob_tcp_msg_recv: peer closed connection
[cn1:13986] [[15051,0],0] tcp_peer_close for [[15051,0],2] sd 17 state CONNECTED
[cn3:21242] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn3:21242] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn3:21242] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn1:13986] [[15051,0],0] tcp:lost connection called for peer [[15051,0],2]
[cn3:21242] mca: base: close: unloading component binomial
[cn3:21242] mca: base: close: component oob closed
[cn3:21242] mca: base: close: unloading component oob
[cn1:13986] [[15051,0],0] route to [[15051,0],2] lost
[cn1:13986] [[15051,0],0] routed_binomial: removing route to child daemon [[15051,0],2]
[cn3:21242] [[15051,0],2] TCP SHUTDOWN
[cn3:21242] [[15051,0],2] RELEASING PEER OBJ [[15051,0],0]
[cn3:21242] [[15051,0],2] CLOSING SOCKET 9
[cn1:13986] [[15051,0],0] route to [[15051,0],2] lost
[cn3:21242] [[15051,0],2] RELEASING PEER OBJ [[15051,0],1]
[cn1:13986] [[15051,0],0] num routes 2
[cn1:13986] [[15051,0],0] num routes 2
[cn3:21242] [[15051,0],2] RELEASING PEER OBJ [[15051,0],3]
[cn3:21242] [[15051,0],2] RELEASING PEER OBJ [[15051,0],4]
[cn3:21242] [[15051,0],2] RELEASING PEER OBJ [[15051,1],4]
[cn3:21242] [[15051,0],2] CLOSING SOCKET 12
[cn3:21242] [[15051,0],2] RELEASING PEER OBJ [[15051,1],5]
[cn3:21242] [[15051,0],2] CLOSING SOCKET 13
[cn3:21242] mca: base: close: component tcp closed
[cn3:21242] mca: base: close: unloading component tcp
[cn3:21242] mca: base: close: unloading component hash
[cn3:21242] sess_dir_cleanup: job session dir does not exist
[cn3:21242] mca: base: close: component slurm closed
[cn3:21242] mca: base: close: unloading component slurm
exiting with status 0
[cn5:13991] sess_dir_finalize: proc session dir does not exist
[cn5:13991] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn5:13991] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn5:13991] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn5:13991] [[15051,0],4] num routes 0
[cn5:13991] [[15051,0],4]:tcp:recv:handler called for peer [[15051,1],8]
[cn5:13991] [[15051,0],4]:tcp:recv:handler CONNECTED
[cn5:13991] [[15051,0],4]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4]-[[15051,1],8] mca_oob_tcp_msg_recv: peer closed connection
[cn5:13991] [[15051,0],4] tcp_peer_close for [[15051,1],8] sd 12 state CONNECTED
[cn5:13991] [[15051,0],4] tcp:lost connection called for peer [[15051,1],8]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] route to [[15051,1],8] lost
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn5:13991] sess_dir_finalize: proc session dir does not exist
[cn5:13991] [[15051,0],4] rml_send_buffer to peer [[15051,0],0] at tag 5
[cn5:13991] [[15051,0],4] num routes 0
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] rml_send_msg to peer [[15051,0],0] at tag 5
[cn5:13991] [[15051,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] oob:base:send to target [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:base:send known transport for peer [[15051,0],0]
[cn5:13991] [[15051,0],4] oob:tcp:send_nb to peer [[15051,0],0]:5
[cn5:13991] [[15051,0],4] tcp:send_nb to peer [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:5
[cn5:13991] [[15051,0],4] routing direct to the HNP
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 73
[cn5:13991] [[15051,0],4] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn5:13991] [[15051,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler called to send to peer [[15051,0],0]
[cn5:13991] [[15051,0],4] tcp:send_handler SENDING TO [[15051,0],0]
[cn5:13991] [[15051,0],4] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 73 BYTES ON SOCKET 9
[cn5:13991] [[15051,0],4]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:13991] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn5:13991] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn5:13991] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn5:13991] [[15051,0],4] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 21
[cn5:13991] [[15051,0],4] grpcomm:base:receive stop comm
[cn5:13991] [[15051,0],4] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 30
[cn5:13991] [[15051,0],4] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 15
[cn5:13991] [[15051,0],4] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 32
[cn5:13991] mca: base: close: component bad closed
[cn5:13991] mca: base: close: unloading component bad
[cn5:13991] [[15051,0],4] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 3
[cn5:13991] [[15051,0],4] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 45
[cn5:13991] [[15051,0],4] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 46
[cn5:13991] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn5:13991] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn5:13991] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn5:13991] mca: base: close: unloading component binomial
[cn5:13991] mca: base: close: component oob closed
[cn5:13991] mca: base: close: unloading component oob
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],4] (ORIGIN [[15051,0],4]) OF 73 BYTES FOR DEST [[15051,0],0] TAG 5
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:13986] [[15051,0],0] message received 73 bytes from [[15051,0],4] for tag 5
[cn1:13986] [[15051,0],0] plm:base:receive processing msg
[cn1:13986] [[15051,0],0] plm:base:receive update proc state command from [[15051,0],4]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for job [15051,1]
[cn5:13991] [[15051,0],4] TCP SHUTDOWN
[cn5:13991] [[15051,0],4] RELEASING PEER OBJ [[15051,0],0]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 8 state NORMALLY TERMINATED exit_code 11
[cn5:13991] [[15051,0],4] CLOSING SOCKET 9
[cn1:13986] [[15051,0],0] plm:base:receive done processing commands
[cn5:13991] [[15051,0],4] RELEASING PEER OBJ [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],4]
[cn5:13991] [[15051,0],4] RELEASING PEER OBJ [[15051,0],2]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn5:13991] [[15051,0],4] RELEASING PEER OBJ [[15051,0],3]
[cn5:13991] [[15051,0],4] RELEASING PEER OBJ [[15051,1],8]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn5:13991] [[15051,0],4] CLOSING SOCKET 12
[cn1:13986] [[15051,0],0]-[[15051,0],4] mca_oob_tcp_msg_recv: peer closed connection
[cn5:13991] [[15051,0],4] RELEASING PEER OBJ [[15051,1],9]
[cn5:13991] [[15051,0],4] CLOSING SOCKET 13
[cn1:13986] [[15051,0],0] tcp_peer_close for [[15051,0],4] sd 14 state CONNECTED
[cn1:13986] [[15051,0],0] tcp:lost connection called for peer [[15051,0],4]
[cn5:13991] mca: base: close: component tcp closed
[cn5:13991] mca: base: close: unloading component tcp
[cn1:13986] [[15051,0],0] route to [[15051,0],4] lost
[cn1:13986] [[15051,0],0] routed_binomial: removing route to child daemon [[15051,0],4]
[cn1:13986] [[15051,0],0] route to [[15051,0],4] lost
[cn1:13986] [[15051,0],0] num routes 1
[cn1:13986] [[15051,0],0] num routes 1
[cn5:13991] mca: base: close: unloading component hash
[cn5:13991] sess_dir_cleanup: job session dir does not exist
[cn5:13991] mca: base: close: component slurm closed
[cn5:13991] mca: base: close: unloading component slurm
exiting with status 0
[cn2:00785] sess_dir_finalize: proc session dir does not exist
[cn2:00785] sess_dir_finalize: proc session dir does not exist
[cn2:00785] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn2:00785] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn2:00785] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn2:00785] [[15051,0],1] num routes 1
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],3]:15
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],3]) --> [[15051,0],3]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],3] - queueing for send
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,0],1]
[cn4:17654] [[15051,0],3]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]:tcp:recv:handler allocate data region of size 8
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],3]
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,0],3]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 708 BYTES FOR DEST [[15051,0],1] TAG 32
[cn2:00785] [[15051,0],1] DELIVERING TO RML
[cn2:00785] [[15051,0],1] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],3]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],3]
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],3] OF 8 BYTES ON SOCKET 10
[cn2:00785] [[15051,0],1]-[[15051,0],3] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] [[15051,0],3]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 8 BYTES FOR DEST [[15051,0],3] TAG 15
[cn4:17654] [[15051,0],3] DELIVERING TO RML
[cn4:17654] [[15051,0],3] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,1],2]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1]-[[15051,1],2] mca_oob_tcp_msg_recv: peer closed connection
[cn2:00785] [[15051,0],1] tcp_peer_close for [[15051,1],2] sd 13 state CONNECTED
[cn4:17654] [[15051,0],3] message received 8 bytes from [[15051,0],1] for tag 15
[cn4:17654] [[15051,0],3] grpcomm:xcast:recv: with 8 bytes
[cn4:17654] [[15051,0],3] orte:daemon:send_relay - recipient list is empty!
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],3] at tag 1
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],3] at tag 1
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,1],3]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1]-[[15051,1],3] mca_oob_tcp_msg_recv: peer closed connection
[cn4:17654] [[15051,0],3] rml_send_iovec_to_self at tag 1
[cn4:17654] [[15051,0],3] rml_send_to_self callback executing for tag 1
[cn4:17654] [[15051,0],3] message received 8 bytes from [[15051,0],3] for tag 1
[cn2:00785] [[15051,0],1] tcp_peer_close for [[15051,1],3] sd 14 state CONNECTED
[cn4:17654] [[15051,0],3] orted:comm:process_commands() Processing Command: ORTE_DAEMON_EXIT_CMD
[cn4:17654] [[15051,0],3] orted_cmd: received exit cmd
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 1384
[cn2:00785] [[15051,0],1] message received 708 bytes from [[15051,0],3] for tag 32
[cn2:00785] [[15051,0],1] grpcomm:base:daemon_coll: daemon collective recvd from [[15051,0],3]
[cn2:00785] [[15051,0],1] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn2:00785] [[15051,0],1] grpcomm:base:daemon_coll: NUM CONTRIBS: 2
[cn2:00785] [[15051,0],1] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[15051,0],0]
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 32
[cn2:00785] [[15051,0],1] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 10 ngr 4
[cn2:00785] [[15051,0],1] tcp:lost connection called for peer [[15051,1],2]
[cn2:00785] [[15051,0],1] route to [[15051,1],2] lost
[cn2:00785] [[15051,0],1] tcp:lost connection called for peer [[15051,1],3]
[cn2:00785] [[15051,0],1] route to [[15051,1],3] lost
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 32
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:32
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 1384 BYTES FOR DEST [[15051,0],0] TAG 32
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:32
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn1:13986] [[15051,0],0] message received 1384 bytes from [[15051,0],1] for tag 32
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[15051,0],1]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 1384 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 4
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] grpcomm:base:daemon_coll: CANNOT PROGRESS
[cn2:00785] sess_dir_finalize: proc session dir does not exist
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn2:00785] sess_dir_finalize: proc session dir does not exist
[cn2:00785] [[15051,0],1] rml_send_buffer to peer [[15051,0],0] at tag 5
[cn2:00785] [[15051,0],1] num routes 1
[cn2:00785] [[15051,0],1] rml_send_msg to peer [[15051,0],0] at tag 5
[cn2:00785] [[15051,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00785] [[15051,0],1] oob:base:send to target [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:base:send known transport for peer [[15051,0],0]
[cn2:00785] [[15051,0],1] oob:tcp:send_nb to peer [[15051,0],0]:5
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] tcp:send_nb to peer [[15051,0],0]
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:5
[cn2:00785] [[15051,0],1] routing direct to the HNP
[cn2:00785] [[15051,0],1] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],1] (ORIGIN [[15051,0],1]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 5
[cn2:00785] [[15051,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn2:00785] [[15051,0],1] tcp:send_handler called to send to peer [[15051,0],0]
[cn2:00785] [[15051,0],1] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00785] [[15051,0],1] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],1] for tag 5
[cn2:00785] [[15051,0],1]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:13986] [[15051,0],0] plm:base:receive processing msg
[cn1:13986] [[15051,0],0] plm:base:receive update proc state command from [[15051,0],1]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for job [15051,1]
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 2 state NORMALLY TERMINATED exit_code 11
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 3 state NORMALLY TERMINATED exit_code 11
[cn1:13986] [[15051,0],0] plm:base:receive done processing commands
[cn4:17654] sess_dir_finalize: proc session dir does not exist
[cn4:17654] sess_dir_finalize: proc session dir does not exist
[cn4:17654] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn4:17654] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn4:17654] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn4:17654] [[15051,0],3] num routes 0
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,1],6]
[cn4:17654] [[15051,0],3]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]-[[15051,1],6] mca_oob_tcp_msg_recv: peer closed connection
[cn4:17654] [[15051,0],3] tcp_peer_close for [[15051,1],6] sd 13 state CONNECTED
[cn4:17654] [[15051,0],3]:tcp:recv:handler called for peer [[15051,1],7]
[cn4:17654] [[15051,0],3]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3]:tcp:recv:handler allocate new recv msg
[cn4:17654] [[15051,0],3]:tcp:recv:handler read hdr
[cn4:17654] [[15051,0],3]-[[15051,1],7] mca_oob_tcp_msg_recv: peer closed connection
[cn4:17654] [[15051,0],3] tcp_peer_close for [[15051,1],7] sd 14 state CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate data region of size 117
[cn4:17654] [[15051,0],3] tcp:lost connection called for peer [[15051,1],6]
[cn4:17654] [[15051,0],3] route to [[15051,1],6] lost
[cn4:17654] [[15051,0],3] tcp:lost connection called for peer [[15051,1],7]
[cn4:17654] [[15051,0],3] route to [[15051,1],7] lost
[cn4:17654] sess_dir_finalize: proc session dir does not exist
[cn4:17654] sess_dir_finalize: proc session dir does not exist
[cn4:17654] [[15051,0],3] rml_send_buffer to peer [[15051,0],0] at tag 5
[cn4:17654] [[15051,0],3] num routes 0
[cn4:17654] [[15051,0],3] rml_send_msg to peer [[15051,0],0] at tag 5
[cn4:17654] [[15051,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17654] [[15051,0],3] oob:base:send to target [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:base:send known transport for peer [[15051,0],0]
[cn4:17654] [[15051,0],3] oob:tcp:send_nb to peer [[15051,0],0]:5
[cn4:17654] [[15051,0],3] tcp:send_nb to peer [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15051,0],0]
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15051,0],0]:5
[cn4:17654] [[15051,0],3] routing direct to the HNP
[cn4:17654] [[15051,0],3] routed_binomial_get([[15051,0],0]) --> [[15051,0],0]
[cn4:17654] [[15051,0],3] tcp:send_nb: already connected to [[15051,0],0] - queueing for send
[cn4:17654] [[15051,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15051,0],0]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn4:17654] [[15051,0],3] tcp:send_handler called to send to peer [[15051,0],0]
[cn1:13986] [[15051,0],0] RECVD COMPLETE MESSAGE FROM [[15051,0],3] (ORIGIN [[15051,0],3]) OF 117 BYTES FOR DEST [[15051,0],0] TAG 5
[cn4:17654] [[15051,0],3] tcp:send_handler SENDING TO [[15051,0],0]
[cn1:13986] [[15051,0],0] DELIVERING TO RML
[cn4:17654] [[15051,0],3] MESSAGE SEND COMPLETE TO [[15051,0],0] OF 117 BYTES ON SOCKET 9
[cn1:13986] [[15051,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17654] [[15051,0],3]-[[15051,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17654] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn4:17654] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn1:13986] [[15051,0],0] message received 117 bytes from [[15051,0],3] for tag 5
[cn4:17654] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn1:13986] [[15051,0],0] plm:base:receive processing msg
[cn4:17654] [[15051,0],3] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 21
[cn1:13986] [[15051,0],0] plm:base:receive update proc state command from [[15051,0],3]
[cn4:17654] [[15051,0],3] grpcomm:base:receive stop comm
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for job [15051,1]
[cn4:17654] [[15051,0],3] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 30
[cn4:17654] [[15051,0],3] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 15
[cn4:17654] [[15051,0],3] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 32
[cn4:17654] mca: base: close: component bad closed
[cn4:17654] mca: base: close: unloading component bad
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 6 state NORMALLY TERMINATED exit_code 11
[cn4:17654] [[15051,0],3] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 3
[cn1:13986] [[15051,0],0] plm:base:receive got update_proc_state for vpid 7 state NORMALLY TERMINATED exit_code 11
[cn1:13986] [[15051,0],0] plm:base:receive done processing commands
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn1:13986] [[15051,0],0]-[[15051,0],3] mca_oob_tcp_msg_recv: peer closed connection
[cn1:13986] [[15051,0],0] tcp_peer_close for [[15051,0],3] sd 16 state CONNECTED
[cn1:13986] [[15051,0],0] tcp:lost connection called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0] route to [[15051,0],3] lost
[cn1:13986] [[15051,0],0] route to [[15051,0],3] lost
[cn1:13986] [[15051,0],0] num routes 1
[cn1:13986] [[15051,0],0] num routes 1
[cn4:17654] [[15051,0],3] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 45
[cn4:17654] [[15051,0],3] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 46
[cn4:17654] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn4:17654] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn4:17654] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn4:17654] mca: base: close: unloading component binomial
[cn4:17654] mca: base: close: component oob closed
[cn4:17654] mca: base: close: unloading component oob
[cn4:17654] [[15051,0],3] TCP SHUTDOWN
[cn4:17654] [[15051,0],3] RELEASING PEER OBJ [[15051,0],0]
[cn4:17654] [[15051,0],3] CLOSING SOCKET 9
[cn4:17654] [[15051,0],3] RELEASING PEER OBJ [[15051,0],1]
[cn4:17654] [[15051,0],3] CLOSING SOCKET 10
[cn4:17654] [[15051,0],3] RELEASING PEER OBJ [[15051,0],2]
[cn4:17654] [[15051,0],3] RELEASING PEER OBJ [[15051,0],4]
[cn4:17654] [[15051,0],3] RELEASING PEER OBJ [[15051,1],6]
[cn4:17654] [[15051,0],3] CLOSING SOCKET 13
[cn4:17654] [[15051,0],3] RELEASING PEER OBJ [[15051,1],7]
[cn4:17654] [[15051,0],3] CLOSING SOCKET 14
[cn4:17654] mca: base: close: component tcp closed
[cn4:17654] mca: base: close: unloading component tcp
[cn2:00785] [[15051,0],1]:tcp:recv:handler called for peer [[15051,0],3]
[cn2:00785] [[15051,0],1]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1]-[[15051,0],3] mca_oob_tcp_msg_recv: peer closed connection
[cn2:00785] [[15051,0],1] tcp_peer_close for [[15051,0],3] sd 10 state CONNECTED
[cn2:00785] [[15051,0],1] tcp:lost connection called for peer [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler called for peer [[15051,0],1]
[cn4:17654] mca: base: close: unloading component hash
[cn1:13986] [[15051,0],0]:tcp:recv:handler CONNECTED
[cn2:00785] [[15051,0],1] route to [[15051,0],3] lost
[cn1:13986] [[15051,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00785] [[15051,0],1] routed_binomial: removing route to child daemon [[15051,0],3]
[cn1:13986] [[15051,0],0]:tcp:recv:handler read hdr
[cn2:00785] [[15051,0],1] num routes 0
[cn1:13986] [[15051,0],0]-[[15051,0],1] mca_oob_tcp_msg_recv: peer closed connection
[cn2:00785] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn2:00785] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn2:00785] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn1:13986] [[15051,0],0] tcp_peer_close for [[15051,0],1] sd 15 state CONNECTED
[cn2:00785] [[15051,0],1] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 21
[cn1:13986] [[15051,0],0] tcp:lost connection called for peer [[15051,0],1]
[cn2:00785] [[15051,0],1] grpcomm:base:receive stop comm
[cn2:00785] [[15051,0],1] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 30
[cn1:13986] [[15051,0],0] route to [[15051,0],1] lost
[cn2:00785] [[15051,0],1] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 15
[cn1:13986] [[15051,0],0] routed_binomial: removing route to child daemon [[15051,0],1]
[cn2:00785] [[15051,0],1] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 32
[cn2:00785] mca: base: close: component bad closed
[cn2:00785] mca: base: close: unloading component bad
[cn2:00785] [[15051,0],1] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 3
[cn1:13986] [[15051,0],0] route to [[15051,0],1] lost
[cn2:00785] [[15051,0],1] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 45
[cn2:00785] [[15051,0],1] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 46
[cn1:13986] [[15051,0],0] num routes 0
[cn2:00785] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn2:00785] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn2:00785] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn4:17654] sess_dir_cleanup: job session dir does not exist
[cn2:00785] mca: base: close: unloading component binomial
[cn4:17654] mca: base: close: component slurm closed
[cn4:17654] mca: base: close: unloading component slurm
[cn2:00785] mca: base: close: component oob closed
[cn2:00785] mca: base: close: unloading component oob
[cn2:00785] [[15051,0],1] TCP SHUTDOWN
[cn2:00785] [[15051,0],1] RELEASING PEER OBJ [[15051,0],0]
[cn2:00785] [[15051,0],1] CLOSING SOCKET 9
[cn2:00785] [[15051,0],1] RELEASING PEER OBJ [[15051,0],2]
[cn2:00785] [[15051,0],1] RELEASING PEER OBJ [[15051,0],3]
[cn2:00785] [[15051,0],1] CLOSING SOCKET 10
[cn2:00785] [[15051,0],1] RELEASING PEER OBJ [[15051,0],4]
[cn2:00785] [[15051,0],1] RELEASING PEER OBJ [[15051,1],2]
[cn2:00785] [[15051,0],1] CLOSING SOCKET 13
[cn2:00785] [[15051,0],1] RELEASING PEER OBJ [[15051,1],3]
[cn2:00785] [[15051,0],1] CLOSING SOCKET 14
[cn2:00785] mca: base: close: component tcp closed
[cn2:00785] mca: base: close: unloading component tcp
[cn2:00785] mca: base: close: unloading component hash
[cn2:00785] sess_dir_cleanup: job session dir does not exist
[cn2:00785] mca: base: close: component slurm closed
[cn2:00785] mca: base: close: unloading component slurm
exiting with status 0
[cn1:13986] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn1:13986] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn1:13986] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn1:13986] 1 more process has sent help message help-mpi-btl-usnic.txt / internal error after init
[cn1:13986] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[cn1:13986] [[15051,0],0] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 34
[cn1:13986] [[15051,0],0] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 45
[cn1:13986] [[15051,0],0] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 46
[cn1:13986] [[15051,0],0] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 21
[cn1:13986] [[15051,0],0] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 22
[cn1:13986] [[15051,0],0] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 2
exiting with status 0
[cn1:13986] mca: base: close: unloading component lama
[cn1:13986] mca: base: close: component mindist closed
[cn1:13986] mca: base: close: unloading component mindist
[cn1:13986] mca: base: close: component ppr closed
[cn1:13986] mca: base: close: unloading component ppr
[cn1:13986] mca: base: close: component rank_file closed
[cn1:13986] mca: base: close: unloading component rank_file
[cn1:13986] mca: base: close: component resilient closed
[cn1:13986] mca: base: close: unloading component resilient
[cn1:13986] mca: base: close: component round_robin closed
[cn1:13986] mca: base: close: unloading component round_robin
[cn1:13986] mca: base: close: component seq closed
[cn1:13986] mca: base: close: unloading component seq
[cn1:13986] mca: base: close: component staged closed
[cn1:13986] mca: base: close: unloading component staged
[cn1:13986] mca: base: close: component slurm closed
[cn1:13986] mca: base: close: unloading component slurm
[cn1:13986] [[15051,0],0] grpcomm:base:receive stop comm
[cn1:13986] [[15051,0],0] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 30
[cn1:13986] [[15051,0],0] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 15
[cn1:13986] [[15051,0],0] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 32
[cn1:13986] [[15051,0],0] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 33
[cn1:13986] [[15051,0],0] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 48
[cn1:13986] mca: base: close: component bad closed
[cn1:13986] mca: base: close: unloading component bad
[cn1:13986] mca: base: close: unloading component hash
[cn1:13986] mca: base: close: unloading component binomial
[cn1:13986] [[15051,0],0] plm:base:receive stop comm
[cn1:13986] [[15051,0],0] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 5
[cn1:13986] [[15051,0],0] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 10
[cn1:13986] mca: base: close: component slurm closed
[cn1:13986] mca: base: close: unloading component slurm
[cn1:13986] mca: base: close: component oob closed
[cn1:13986] mca: base: close: unloading component oob
[cn1:13986] [[15051,0],0] TCP SHUTDOWN
[cn1:13986] [[15051,0],0] RELEASING PEER OBJ [[15051,0],4]
[cn1:13986] [[15051,0],0] CLOSING SOCKET 14
[cn1:13986] [[15051,0],0] RELEASING PEER OBJ [[15051,0],1]
[cn1:13986] [[15051,0],0] CLOSING SOCKET 15
[cn1:13986] [[15051,0],0] RELEASING PEER OBJ [[15051,0],3]
[cn1:13986] [[15051,0],0] CLOSING SOCKET 16
[cn1:13986] [[15051,0],0] RELEASING PEER OBJ [[15051,0],2]
[cn1:13986] [[15051,0],0] CLOSING SOCKET 17
[cn1:13986] [[15051,0],0] RELEASING PEER OBJ [[15051,1],0]
[cn1:13986] [[15051,0],0] CLOSING SOCKET 20
[cn1:13986] [[15051,0],0] RELEASING PEER OBJ [[15051,1],1]
[cn1:13986] [[15051,0],0] CLOSING SOCKET 23
[cn1:13986] mca: base: close: component tcp closed
[cn1:13986] mca: base: close: unloading component tcp
[cn1:13986] sess_dir_cleanup: job session dir does not exist
[cn1:13986] mca: base: close: component hnp closed
[cn1:13986] mca: base: close: unloading component hnp
exiting with status 1
