[cn1:14018] progress: initialized event flag to: 3
[cn1:14018] progress: initialized yield_when_idle to: false
[cn1:14018] progress: initialized num users to: 0
[cn1:14018] progress: initialized poll rate to: 23940000
[cn1:14018] progress: event_users_increment setting count to 1
[cn1:14018] mca: base: components_register: registering ess components
[cn1:14018] mca: base: components_register: found loaded component env
[cn1:14018] mca: base: components_register: component env has no register or open function
[cn1:14018] mca: base: components_register: found loaded component hnp
[cn1:14018] mca: base: components_register: component hnp has no register or open function
[cn1:14018] mca: base: components_register: found loaded component pmi
[cn1:14018] mca: base: components_register: component pmi has no register or open function
[cn1:14018] mca: base: components_register: found loaded component singleton
[cn1:14018] mca: base: components_register: component singleton register function successful
[cn1:14018] mca: base: components_register: found loaded component slurm
[cn1:14018] mca: base: components_register: component slurm has no register or open function
[cn1:14018] mca: base: components_register: found loaded component tool
[cn1:14018] mca: base: components_register: component tool has no register or open function
[cn1:14018] mca: base: components_open: opening ess components
[cn1:14018] mca: base: components_open: found loaded component env
[cn1:14018] mca: base: components_open: component env open function successful
[cn1:14018] mca: base: components_open: found loaded component hnp
[cn1:14018] mca: base: components_open: component hnp open function successful
[cn1:14018] mca: base: components_open: found loaded component pmi
[cn1:14018] mca: base: components_open: component pmi open function successful
[cn1:14018] mca: base: components_open: found loaded component singleton
[cn1:14018] mca: base: components_open: component singleton open function successful
[cn1:14018] mca: base: components_open: found loaded component slurm
[cn1:14018] mca: base: components_open: component slurm open function successful
[cn1:14018] mca: base: components_open: found loaded component tool
[cn1:14018] mca: base: components_open: component tool open function successful
[cn1:14018] mca:base:select: Auto-selecting ess components
[cn1:14018] mca:base:select:(  ess) Querying component [env]
[cn1:14018] mca:base:select:(  ess) Skipping component [env]. Query failed to return a module
[cn1:14018] mca:base:select:(  ess) Querying component [hnp]
[cn1:14018] mca:base:select:(  ess) Query of component [hnp] set priority to 100
[cn1:14018] mca:base:select:(  ess) Querying component [pmi]
[cn1:14018] mca:base:select:(  ess) Skipping component [pmi]. Query failed to return a module
[cn1:14018] mca:base:select:(  ess) Querying component [singleton]
[cn1:14018] mca:base:select:(  ess) Skipping component [singleton]. Query failed to return a module
[cn1:14018] mca:base:select:(  ess) Querying component [slurm]
[cn1:14018] mca:base:select:(  ess) Skipping component [slurm]. Query failed to return a module
[cn1:14018] mca:base:select:(  ess) Querying component [tool]
[cn1:14018] mca:base:select:(  ess) Skipping component [tool]. Query failed to return a module
[cn1:14018] mca:base:select:(  ess) Selected component [hnp]
[cn1:14018] mca: base: close: component env closed
[cn1:14018] mca: base: close: unloading component env
[cn1:14018] mca: base: close: component pmi closed
[cn1:14018] mca: base: close: unloading component pmi
[cn1:14018] mca: base: close: component singleton closed
[cn1:14018] mca: base: close: unloading component singleton
[cn1:14018] mca: base: close: component slurm closed
[cn1:14018] mca: base: close: unloading component slurm
[cn1:14018] mca: base: close: component tool closed
[cn1:14018] mca: base: close: unloading component tool
[cn1:14018] [[INVALID],INVALID] Topology Info:
[cn1:14018] Type: Machine Number of child objects: 2
	Name=NULL
	total=25100516KB
	DMIProductName=S5520UR
	DMIProductVersion=....................
	DMIBoardVendor="Intel Corporation"
	DMIBoardName=S5520UR
	DMIBoardVersion=E22554-751
	DMIBoardAssetTag=....................
	DMIChassisVendor=...............................
	DMIChassisType=23
	DMIChassisVersion=..................
	DMIChassisAssetTag=....................
	DMIBIOSVendor="Intel Corp."
	DMIBIOSVersion=S5500.86B.01.00.0054.092820101104
	DMIBIOSDate=09/28/2010
	DMISysVendor="Intel Corporation"
	Backend=Linux
	LinuxCgroup=/
	OSName=Linux
	OSRelease=2.6.32-431.3.1.el6.x86_64
	OSVersion="#1 SMP Fri Jan 3 21:39:27 UTC 2014"
	Architecture=x86_64
	Cpuset:  0x000000ff
	Online:  0x000000ff
	Allowed: 0x000000ff
	Bind CPU proc:   TRUE
	Bind CPU thread: TRUE
	Bind MEM proc:   FALSE
	Bind MEM thread: FALSE
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12517604KB
		total=12517604KB
		Cpuset:  0x00000055
		Online:  0x00000055
		Allowed: 0x00000055
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x00000055
			Online:  0x00000055
			Allowed: 0x00000055
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x00000055
				Online:  0x00000055
				Allowed: 0x00000055
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000001
					Online:  0x00000001
					Allowed: 0x00000001
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000001
						Online:  0x00000001
						Allowed: 0x00000001
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000001
							Online:  0x00000001
							Allowed: 0x00000001
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000001
								Online:  0x00000001
								Allowed: 0x00000001
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000004
					Online:  0x00000004
					Allowed: 0x00000004
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000004
						Online:  0x00000004
						Allowed: 0x00000004
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000004
							Online:  0x00000004
							Allowed: 0x00000004
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000004
								Online:  0x00000004
								Allowed: 0x00000004
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000010
					Online:  0x00000010
					Allowed: 0x00000010
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000010
						Online:  0x00000010
						Allowed: 0x00000010
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000010
							Online:  0x00000010
							Allowed: 0x00000010
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000010
								Online:  0x00000010
								Allowed: 0x00000010
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000040
					Online:  0x00000040
					Allowed: 0x00000040
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000040
						Online:  0x00000040
						Allowed: 0x00000040
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000040
							Online:  0x00000040
							Allowed: 0x00000040
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000040
								Online:  0x00000040
								Allowed: 0x00000040
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12582912KB
		total=12582912KB
		Cpuset:  0x000000aa
		Online:  0x000000aa
		Allowed: 0x000000aa
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x000000aa
			Online:  0x000000aa
			Allowed: 0x000000aa
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x000000aa
				Online:  0x000000aa
				Allowed: 0x000000aa
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000002
					Online:  0x00000002
					Allowed: 0x00000002
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000002
						Online:  0x00000002
						Allowed: 0x00000002
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000002
							Online:  0x00000002
							Allowed: 0x00000002
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000002
								Online:  0x00000002
								Allowed: 0x00000002
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000008
					Online:  0x00000008
					Allowed: 0x00000008
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000008
						Online:  0x00000008
						Allowed: 0x00000008
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000008
							Online:  0x00000008
							Allowed: 0x00000008
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000008
								Online:  0x00000008
								Allowed: 0x00000008
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000020
					Online:  0x00000020
					Allowed: 0x00000020
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000020
						Online:  0x00000020
						Allowed: 0x00000020
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000020
							Online:  0x00000020
							Allowed: 0x00000020
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000020
								Online:  0x00000020
								Allowed: 0x00000020
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000080
					Online:  0x00000080
					Allowed: 0x00000080
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000080
						Online:  0x00000080
						Allowed: 0x00000080
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000080
							Online:  0x00000080
							Allowed: 0x00000080
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000080
								Online:  0x00000080
								Allowed: 0x00000080
[cn1:14018] mca: base: components_register: registering plm components
[cn1:14018] mca: base: components_register: found loaded component isolated
[cn1:14018] mca: base: components_register: component isolated has no register or open function
[cn1:14018] mca: base: components_register: found loaded component rsh
[cn1:14018] mca: base: components_register: component rsh register function successful
[cn1:14018] mca: base: components_register: found loaded component slurm
[cn1:14018] mca: base: components_register: component slurm register function successful
[cn1:14018] mca: base: components_open: opening plm components
[cn1:14018] mca: base: components_open: found loaded component isolated
[cn1:14018] mca: base: components_open: component isolated open function successful
[cn1:14018] mca: base: components_open: found loaded component rsh
[cn1:14018] mca: base: components_open: component rsh open function successful
[cn1:14018] mca: base: components_open: found loaded component slurm
[cn1:14018] mca: base: components_open: component slurm open function successful
[cn1:14018] mca:base:select: Auto-selecting plm components
[cn1:14018] mca:base:select:(  plm) Querying component [isolated]
[cn1:14018] mca:base:select:(  plm) Query of component [isolated] set priority to 0
[cn1:14018] mca:base:select:(  plm) Querying component [rsh]
[cn1:14018] [[INVALID],INVALID] plm:rsh_lookup on agent ssh : rsh path NULL
[cn1:14018] mca:base:select:(  plm) Query of component [rsh] set priority to 10
[cn1:14018] mca:base:select:(  plm) Querying component [slurm]
[cn1:14018] [[INVALID],INVALID] plm:slurm: available for selection
[cn1:14018] mca:base:select:(  plm) Query of component [slurm] set priority to 75
[cn1:14018] mca:base:select:(  plm) Selected component [slurm]
[cn1:14018] mca: base: close: component isolated closed
[cn1:14018] mca: base: close: unloading component isolated
[cn1:14018] mca: base: close: component rsh closed
[cn1:14018] mca: base: close: unloading component rsh
[cn1:14018] plm:base:set_hnp_name: initial bias 14018 nodename hash 1752458269
[cn1:14018] plm:base:set_hnp_name: final jobfam 15019
[cn1:14018] mca: base: components_register: registering oob components
[cn1:14018] mca: base: components_register: found loaded component tcp
[cn1:14018] mca: base: components_register: component tcp register function successful
[cn1:14018] mca: base: components_open: opening oob components
[cn1:14018] mca: base: components_open: found loaded component tcp
[cn1:14018] mca: base: components_open: component tcp open function successful
[cn1:14018] mca:oob:select: checking available component tcp
[cn1:14018] mca:oob:select: Querying component [tcp]
[cn1:14018] oob:tcp: component_available called
[cn1:14018] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn1:14018] [[15019,0],0] oob:tcp:init rejecting loopback interface lo
[cn1:14018] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn1:14018] [[15019,0],0] oob:tcp:init creating module for V4 address on interface eth0
[cn1:14018] [[15019,0],0] creating OOB-TCP module for interface eth0
[cn1:14018] [[15019,0],0] oob:tcp:init adding 192.168.1.1 to our list of V4 connections
[cn1:14018] WORKING INTERFACE 3 KERNEL INDEX 3 FAMILY: V4
[cn1:14018] [[15019,0],0] oob:tcp:init creating module for V4 address on interface eth1
[cn1:14018] [[15019,0],0] creating OOB-TCP module for interface eth1
[cn1:14018] [[15019,0],0] oob:tcp:init adding 192.168.10.2 to our list of V4 connections
[cn1:14018] [[15019,0],0] TCP STARTUP
[cn1:14018] [[15019,0],0] attempting to bind to IPv4 port 0
[cn1:14018] [[15019,0],0] assigned IPv4 port 57056
[cn1:14018] mca:oob:select: Adding component to end
[cn1:14018] mca:oob:select: Found 1 active transports
[cn1:14018] mca: base: components_register: registering rml components
[cn1:14018] mca: base: components_register: found loaded component oob
[cn1:14018] mca: base: components_register: component oob has no register or open function
[cn1:14018] mca: base: components_open: opening rml components
[cn1:14018] mca: base: components_open: found loaded component oob
[cn1:14018] mca: base: components_open: component oob open function successful
[cn1:14018] orte_rml_base_select: initializing rml component oob
[cn1:14018] mca: base: components_register: registering routed components
[cn1:14018] mca: base: components_register: found loaded component binomial
[cn1:14018] mca: base: components_register: component binomial has no register or open function
[cn1:14018] mca: base: components_register: found loaded component debruijn
[cn1:14018] mca: base: components_register: component debruijn has no register or open function
[cn1:14018] mca: base: components_register: found loaded component direct
[cn1:14018] mca: base: components_register: component direct has no register or open function
[cn1:14018] mca: base: components_register: found loaded component radix
[cn1:14018] mca: base: components_register: component radix register function successful
[cn1:14018] mca: base: components_open: opening routed components
[cn1:14018] mca: base: components_open: found loaded component binomial
[cn1:14018] mca: base: components_open: found loaded component debruijn
[cn1:14018] mca: base: components_open: found loaded component direct
[cn1:14018] mca: base: components_open: found loaded component radix
[cn1:14018] mca:base:select: Auto-selecting routed components
[cn1:14018] mca:base:select:(routed) Querying component [binomial]
[cn1:14018] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn1:14018] mca:base:select:(routed) Querying component [debruijn]
[cn1:14018] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn1:14018] mca:base:select:(routed) Querying component [direct]
[cn1:14018] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn1:14018] mca:base:select:(routed) Querying component [radix]
[cn1:14018] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn1:14018] mca:base:select:(routed) Selected component [binomial]
[cn1:14018] mca: base: close: unloading component debruijn
[cn1:14018] mca: base: close: unloading component direct
[cn1:14018] mca: base: close: unloading component radix
[cn1:14018] orte_routed_base_select: initializing selected component binomial
[cn1:14018] mca: base: components_register: registering dstore components
[cn1:14018] mca: base: components_register: found loaded component hash
[cn1:14018] mca: base: components_register: component hash register function successful
[cn1:14018] mca: base: components_open: opening dstore components
[cn1:14018] mca: base: components_open: found loaded component hash
[cn1:14018] mca:dstore:select: checking available component hash
[cn1:14018] mca:dstore:select: Querying component [hash]
[cn1:14018] mca: base: components_register: registering grpcomm components
[cn1:14018] mca: base: components_register: found loaded component bad
[cn1:14018] mca: base: components_register: component bad has no register or open function
[cn1:14018] mca: base: components_register: found loaded component pmi
[cn1:14018] mca: base: components_register: component pmi register function successful
[cn1:14018] mca: base: components_open: opening grpcomm components
[cn1:14018] mca: base: components_open: found loaded component bad
[cn1:14018] mca: base: components_open: component bad open function successful
[cn1:14018] mca: base: components_open: found loaded component pmi
[cn1:14018] mca: base: components_open: component pmi open function successful
[cn1:14018] mca:base:select: Auto-selecting grpcomm components
[cn1:14018] mca:base:select:(grpcomm) Querying component [bad]
[cn1:14018] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn1:14018] mca:base:select:(grpcomm) Querying component [pmi]
[cn1:14018] mca:base:select:(grpcomm) Skipping component [pmi]. Query failed to return a module
[cn1:14018] mca:base:select:(grpcomm) Selected component [bad]
[cn1:14018] mca: base: close: component pmi closed
[cn1:14018] mca: base: close: unloading component pmi
[cn1:14018] [[15019,0],0] grpcomm:base:receive start comm
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 15
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 32
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 33
[cn1:14018] [[15019,0],0] plm:base:receive start comm
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 5
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 10
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 12
[cn1:14018] mca: base: components_register: registering ras components
[cn1:14018] mca: base: components_register: found loaded component loadleveler
[cn1:14018] mca: base: components_register: component loadleveler register function successful
[cn1:14018] mca: base: components_register: found loaded component simulator
[cn1:14018] mca: base: components_register: component simulator register function successful
[cn1:14018] mca: base: components_register: found loaded component slurm
[cn1:14018] mca: base: components_register: component slurm register function successful
[cn1:14018] mca: base: components_open: opening ras components
[cn1:14018] mca: base: components_open: found loaded component loadleveler
[cn1:14018] mca: base: components_open: component loadleveler open function successful
[cn1:14018] mca: base: components_open: found loaded component simulator
[cn1:14018] mca: base: components_open: found loaded component slurm
[cn1:14018] mca: base: components_open: component slurm open function successful
[cn1:14018] mca:base:select: Auto-selecting ras components
[cn1:14018] mca:base:select:(  ras) Querying component [loadleveler]
[cn1:14018] [[15019,0],0] ras:loadleveler: NOT available for selection
[cn1:14018] mca:base:select:(  ras) Skipping component [loadleveler]. Query failed to return a module
[cn1:14018] mca:base:select:(  ras) Querying component [simulator]
[cn1:14018] mca:base:select:(  ras) Skipping component [simulator]. Query failed to return a module
[cn1:14018] mca:base:select:(  ras) Querying component [slurm]
[cn1:14018] [[15019,0],0] ras:slurm: available for selection
[cn1:14018] mca:base:select:(  ras) Query of component [slurm] set priority to 50
[cn1:14018] mca:base:select:(  ras) Selected component [slurm]
[cn1:14018] mca: base: close: unloading component loadleveler
[cn1:14018] mca: base: close: unloading component simulator
[cn1:14018] mca: base: components_register: registering rmaps components
[cn1:14018] mca: base: components_register: found loaded component lama
[cn1:14018] mca:rmaps:lama: Priority   0
[cn1:14018] mca:rmaps:lama: Map   : NULL
[cn1:14018] mca:rmaps:lama: Bind  : NULL
[cn1:14018] mca:rmaps:lama: MPPR  : NULL
[cn1:14018] mca:rmaps:lama: Order : NULL
[cn1:14018] mca: base: components_register: component lama register function successful
[cn1:14018] mca: base: components_register: found loaded component mindist
[cn1:14018] mca: base: components_register: component mindist register function successful
[cn1:14018] mca: base: components_register: found loaded component ppr
[cn1:14018] mca: base: components_register: component ppr register function successful
[cn1:14018] mca: base: components_register: found loaded component rank_file
[cn1:14018] mca: base: components_register: component rank_file register function successful
[cn1:14018] mca: base: components_register: found loaded component resilient
[cn1:14018] mca: base: components_register: component resilient register function successful
[cn1:14018] mca: base: components_register: found loaded component round_robin
[cn1:14018] mca: base: components_register: component round_robin register function successful
[cn1:14018] mca: base: components_register: found loaded component seq
[cn1:14018] mca: base: components_register: component seq register function successful
[cn1:14018] mca: base: components_register: found loaded component staged
[cn1:14018] mca: base: components_register: component staged has no register or open function
[cn1:14018] mca: base: components_open: opening rmaps components
[cn1:14018] mca: base: components_open: found loaded component lama
[cn1:14018] mca: base: components_open: found loaded component mindist
[cn1:14018] mca: base: components_open: component mindist open function successful
[cn1:14018] mca: base: components_open: found loaded component ppr
[cn1:14018] mca: base: components_open: component ppr open function successful
[cn1:14018] mca: base: components_open: found loaded component rank_file
[cn1:14018] mca: base: components_open: component rank_file open function successful
[cn1:14018] mca: base: components_open: found loaded component resilient
[cn1:14018] mca: base: components_open: component resilient open function successful
[cn1:14018] mca: base: components_open: found loaded component round_robin
[cn1:14018] mca: base: components_open: component round_robin open function successful
[cn1:14018] mca: base: components_open: found loaded component seq
[cn1:14018] mca: base: components_open: component seq open function successful
[cn1:14018] mca: base: components_open: found loaded component staged
[cn1:14018] mca: base: components_open: component staged open function successful
[cn1:14018] mca:rmaps:select: checking available component lama
[cn1:14018] mca:rmaps:select: Querying component [lama]
[cn1:14018] mca:rmaps:select: checking available component mindist
[cn1:14018] mca:rmaps:select: Querying component [mindist]
[cn1:14018] mca:rmaps:select: checking available component ppr
[cn1:14018] mca:rmaps:select: Querying component [ppr]
[cn1:14018] mca:rmaps:select: checking available component rank_file
[cn1:14018] mca:rmaps:select: Querying component [rank_file]
[cn1:14018] mca:rmaps:select: checking available component resilient
[cn1:14018] mca:rmaps:select: Querying component [resilient]
[cn1:14018] mca:rmaps:select: checking available component round_robin
[cn1:14018] mca:rmaps:select: Querying component [round_robin]
[cn1:14018] mca:rmaps:select: checking available component seq
[cn1:14018] mca:rmaps:select: Querying component [seq]
[cn1:14018] mca:rmaps:select: checking available component staged
[cn1:14018] mca:rmaps:select: Querying component [staged]
[cn1:14018] [[15019,0],0]: Final mapper priorities
[cn1:14018] 	Mapper: ppr Priority: 90
[cn1:14018] 	Mapper: seq Priority: 60
[cn1:14018] 	Mapper: resilient Priority: 40
[cn1:14018] 	Mapper: mindist Priority: 20
[cn1:14018] 	Mapper: round_robin Priority: 10
[cn1:14018] 	Mapper: staged Priority: 5
[cn1:14018] 	Mapper: lama Priority: 0
[cn1:14018] 	Mapper: rank_file Priority: 0
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 34
[cn1:14018] [[15019,0],0] setting up session dir with
	tmpdir: UNDEF
	host cn1
[cn1:14018] procdir: /tmp/openmpi-sessions-artpol@cn1_0/15019/0/0
[cn1:14018] jobdir: /tmp/openmpi-sessions-artpol@cn1_0/15019/0
[cn1:14018] top: openmpi-sessions-artpol@cn1_0
[cn1:14018] tmp: /tmp
[cn1:14018] sess_dir_cleanup: job session dir does not exist
[cn1:14018] procdir: /tmp/openmpi-sessions-artpol@cn1_0/15019/0/0
[cn1:14018] jobdir: /tmp/openmpi-sessions-artpol@cn1_0/15019/0
[cn1:14018] top: openmpi-sessions-artpol@cn1_0
[cn1:14018] tmp: /tmp
[cn1:14018] [[15019,0],0] writing contact file /tmp/openmpi-sessions-artpol@cn1_0/15019/contact.txt
[cn1:14018] [[15019,0],0] wrote contact file
[cn1:14018] [[15019,0],0] routed_binomial: init routes for HNP job [15019,0]
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 2
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 21
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 22
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 45
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 46
[cn1:14018] progress: progress_set_yield_when_idle to false
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 1
[cn1:14018] [[15019,0],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 27
[cn1:14018] [[15019,0],0] Monitoring debugger attach fifo /tmp/openmpi-sessions-artpol@cn1_0/15019/0/debugger_attach_fifo
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 15 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 32 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 33 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 5 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 10 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 12 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 34 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 2 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 21 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 22 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 45 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 46 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 1 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] posting recv
[cn1:14018] [[15019,0],0] posting persistent recv on tag 27 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] plm:base:setup_job
[cn1:14018] [[15019,0],0] ras:base:allocate
[cn1:14018] [[15019,0],0] ras:slurm:allocate:discover: checking nodelist: cn[1-5]
[cn1:14018] [[15019,0],0] ras:slurm:allocate:discover: parse range 1-5 (2)
[cn1:14018] [[15019,0],0] ras:slurm:allocate:discover: adding node cn1 (2 slots)
[cn1:14018] [[15019,0],0] ras:slurm:allocate:discover: adding node cn2 (2 slots)
[cn1:14018] [[15019,0],0] ras:slurm:allocate:discover: adding node cn3 (2 slots)
[cn1:14018] [[15019,0],0] ras:slurm:allocate:discover: adding node cn4 (2 slots)
[cn1:14018] [[15019,0],0] ras:slurm:allocate:discover: adding node cn5 (2 slots)
[cn1:14018] [[15019,0],0] ras:slurm:allocate: success
[cn1:14018] [[15019,0],0] ras:base:node_insert inserting 5 nodes
[cn1:14018] [[15019,0],0] ras:base:node_insert updating HNP [cn1] info to 2 slots
[cn1:14018] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn1:14018] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn1:14018] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn1:14018] [[15019,0],0] ras:base:node_insert node cn2 slots 2
[cn1:14018] [[15019,0],0] ras:base:node_insert node cn3 slots 2
[cn1:14018] [[15019,0],0] ras:base:node_insert node cn4 slots 2
[cn1:14018] [[15019,0],0] ras:base:node_insert node cn5 slots 2
[cn1:14018] [[15019,0],0] plm:slurm: LAUNCH DAEMONS CALLED
[cn1:14018] [[15019,0],0] plm:base:setup_vm
[cn1:14018] [[15019,0],0] plm:base:setup_vm creating map
[cn1:14018] [[15019,0],0] plm:base:setup_vm add new daemon [[15019,0],1]
[cn1:14018] [[15019,0],0] plm:base:setup_vm assigning new daemon [[15019,0],1] to node cn2
[cn1:14018] [[15019,0],0] plm:base:setup_vm add new daemon [[15019,0],2]
[cn1:14018] [[15019,0],0] plm:base:setup_vm assigning new daemon [[15019,0],2] to node cn3
[cn1:14018] [[15019,0],0] plm:base:setup_vm add new daemon [[15019,0],3]
[cn1:14018] [[15019,0],0] plm:base:setup_vm assigning new daemon [[15019,0],3] to node cn4
[cn1:14018] [[15019,0],0] plm:base:setup_vm add new daemon [[15019,0],4]
[cn1:14018] [[15019,0],0] plm:base:setup_vm assigning new daemon [[15019,0],4] to node cn5
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 0 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial 0 found child 1
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 1 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 0
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 1
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 1 parent 0 me 1 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial 1 found child 3
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 3 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 0
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 1
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 1 parent 0 me 3 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 1
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 3
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 3 parent 1 me 3 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 1
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 1
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 0
[cn1:14018] [[15019,0],0] routed:binomial 0 found child 2
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 2 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 0
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 1
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 1 parent 0 me 2 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 1
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 3
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 3 parent 1 me 2 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 3
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 7
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 5
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 2
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 2 parent 0 me 2 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 0
[cn1:14018] [[15019,0],0] routed:binomial 0 found child 4
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 0
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 1
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 1 parent 0 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 1
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 3
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 3 parent 1 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 3
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 7
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 5
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 2
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 2 parent 0 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 2
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 6
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 4
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 4 parent 0 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 0
[cn1:14018] [[15019,0],0]: parent 0 num_children 3
[cn1:14018] [[15019,0],0]: 	child 1
[cn1:14018] [[15019,0],0]: 		relation 3
[cn1:14018] [[15019,0],0]: 	child 2
[cn1:14018] [[15019,0],0]: 	child 4
[cn1:14018] [[15019,0],0] plm:slurm: launching on nodes cn2,cn3,cn4,cn5
[cn1:14018] [[15019,0],0] plm:slurm: final top-level argv:
	srun --ntasks-per-node=1 --kill-on-bad-exit --cpu_bind=none --nodes=4 --nodelist=cn2,cn3,cn4,cn5 --ntasks=4 orted --debug --debug-daemons -mca orte_ess_jobid 984285184 -mca orte_ess_vpid 1 -mca orte_ess_num_procs 5 -mca orte_hnp_uri "984285184.0;tcp://192.168.1.1,192.168.10.2:57056"
Daemon was launched on cn5 - beginning to initialize
Daemon was launched on cn2 - beginning to initialize
Daemon was launched on cn4 - beginning to initialize
Daemon was launched on cn3 - beginning to initialize
[cn5:14014] progress: initialized event flag to: 3
[cn5:14014] progress: initialized yield_when_idle to: false
[cn5:14014] progress: initialized num users to: 0
[cn5:14014] progress: initialized poll rate to: 23940000
[cn5:14014] progress: event_users_increment setting count to 1
[cn5:14014] mca: base: components_register: registering ess components
[cn5:14014] mca: base: components_register: found loaded component env
[cn5:14014] mca: base: components_register: component env has no register or open function
[cn5:14014] mca: base: components_register: found loaded component hnp
[cn5:14014] mca: base: components_register: component hnp has no register or open function
[cn5:14014] mca: base: components_register: found loaded component pmi
[cn5:14014] mca: base: components_register: component pmi has no register or open function
[cn5:14014] mca: base: components_register: found loaded component singleton
[cn5:14014] mca: base: components_register: component singleton register function successful
[cn5:14014] mca: base: components_register: found loaded component slurm
[cn5:14014] mca: base: components_register: component slurm has no register or open function
[cn5:14014] mca: base: components_register: found loaded component tool
[cn5:14014] mca: base: components_register: component tool has no register or open function
[cn5:14014] mca: base: components_open: opening ess components
[cn5:14014] mca: base: components_open: found loaded component env
[cn5:14014] mca: base: components_open: component env open function successful
[cn5:14014] mca: base: components_open: found loaded component hnp
[cn5:14014] mca: base: components_open: component hnp open function successful
[cn5:14014] mca: base: components_open: found loaded component pmi
[cn5:14014] mca: base: components_open: component pmi open function successful
[cn5:14014] mca: base: components_open: found loaded component singleton
[cn5:14014] mca: base: components_open: component singleton open function successful
[cn5:14014] mca: base: components_open: found loaded component slurm
[cn5:14014] mca: base: components_open: component slurm open function successful
[cn5:14014] mca: base: components_open: found loaded component tool
[cn5:14014] mca: base: components_open: component tool open function successful
[cn5:14014] mca:base:select: Auto-selecting ess components
[cn5:14014] mca:base:select:(  ess) Querying component [env]
[cn5:14014] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn5:14014] mca:base:select:(  ess) Querying component [hnp]
[cn5:14014] mca:base:select:(  ess) Skipping component [hnp]. Query failed to return a module
[cn5:14014] mca:base:select:(  ess) Querying component [pmi]
[cn5:14014] mca:base:select:(  ess) Query of component [pmi] set priority to 35
[cn5:14014] mca:base:select:(  ess) Querying component [singleton]
[cn5:14014] mca:base:select:(  ess) Skipping component [singleton]. Query failed to return a module
[cn5:14014] mca:base:select:(  ess) Querying component [slurm]
[cn5:14014] mca:base:select:(  ess) Query of component [slurm] set priority to 50
[cn5:14014] mca:base:select:(  ess) Querying component [tool]
[cn5:14014] mca:base:select:(  ess) Skipping component [tool]. Query failed to return a module
[cn5:14014] mca:base:select:(  ess) Selected component [slurm]
[cn5:14014] mca: base: close: component env closed
[cn5:14014] mca: base: close: unloading component env
[cn5:14014] mca: base: close: component hnp closed
[cn5:14014] mca: base: close: unloading component hnp
[cn5:14014] mca: base: close: component pmi closed
[cn5:14014] mca: base: close: unloading component pmi
[cn5:14014] mca: base: close: component singleton closed
[cn5:14014] mca: base: close: unloading component singleton
[cn5:14014] mca: base: close: component tool closed
[cn5:14014] mca: base: close: unloading component tool
[cn2:00814] progress: initialized event flag to: 3
[cn2:00814] progress: initialized yield_when_idle to: false
[cn2:00814] progress: initialized num users to: 0
[cn2:00814] progress: initialized poll rate to: 23940000
[cn2:00814] progress: event_users_increment setting count to 1
[cn2:00814] mca: base: components_register: registering ess components
[cn2:00814] mca: base: components_register: found loaded component env
[cn2:00814] mca: base: components_register: component env has no register or open function
[cn2:00814] mca: base: components_register: found loaded component hnp
[cn2:00814] mca: base: components_register: component hnp has no register or open function
[cn2:00814] mca: base: components_register: found loaded component pmi
[cn2:00814] mca: base: components_register: component pmi has no register or open function
[cn2:00814] mca: base: components_register: found loaded component singleton
[cn2:00814] mca: base: components_register: component singleton register function successful
[cn2:00814] mca: base: components_register: found loaded component slurm
[cn2:00814] mca: base: components_register: component slurm has no register or open function
[cn2:00814] mca: base: components_register: found loaded component tool
[cn2:00814] mca: base: components_register: component tool has no register or open function
[cn2:00814] mca: base: components_open: opening ess components
[cn2:00814] mca: base: components_open: found loaded component env
[cn2:00814] mca: base: components_open: component env open function successful
[cn2:00814] mca: base: components_open: found loaded component hnp
[cn2:00814] mca: base: components_open: component hnp open function successful
[cn2:00814] mca: base: components_open: found loaded component pmi
[cn2:00814] mca: base: components_open: component pmi open function successful
[cn2:00814] mca: base: components_open: found loaded component singleton
[cn2:00814] mca: base: components_open: component singleton open function successful
[cn2:00814] mca: base: components_open: found loaded component slurm
[cn2:00814] mca: base: components_open: component slurm open function successful
[cn2:00814] mca: base: components_open: found loaded component tool
[cn2:00814] mca: base: components_open: component tool open function successful
[cn2:00814] mca:base:select: Auto-selecting ess components
[cn2:00814] mca:base:select:(  ess) Querying component [env]
[cn2:00814] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn2:00814] mca:base:select:(  ess) Querying component [hnp]
[cn2:00814] mca:base:select:(  ess) Skipping component [hnp]. Query failed to return a module
[cn2:00814] mca:base:select:(  ess) Querying component [pmi]
[cn2:00814] mca:base:select:(  ess) Query of component [pmi] set priority to 35
[cn2:00814] mca:base:select:(  ess) Querying component [singleton]
[cn2:00814] mca:base:select:(  ess) Skipping component [singleton]. Query failed to return a module
[cn2:00814] mca:base:select:(  ess) Querying component [slurm]
[cn2:00814] mca:base:select:(  ess) Query of component [slurm] set priority to 50
[cn2:00814] mca:base:select:(  ess) Querying component [tool]
[cn2:00814] mca:base:select:(  ess) Skipping component [tool]. Query failed to return a module
[cn2:00814] mca:base:select:(  ess) Selected component [slurm]
[cn2:00814] mca: base: close: component env closed
[cn2:00814] mca: base: close: unloading component env
[cn2:00814] mca: base: close: component hnp closed
[cn2:00814] mca: base: close: unloading component hnp
[cn2:00814] mca: base: close: component pmi closed
[cn2:00814] mca: base: close: unloading component pmi
[cn2:00814] mca: base: close: component singleton closed
[cn2:00814] mca: base: close: unloading component singleton
[cn2:00814] mca: base: close: component tool closed
[cn2:00814] mca: base: close: unloading component tool
[cn2:00814] ess:slurm setting name
[cn2:00814] ess:slurm set name to [[15019,0],1]
[cn2:00814] ess:slurm set nodename to cn2
[cn4:17677] progress: initialized event flag to: 3
[cn4:17677] progress: initialized yield_when_idle to: false
[cn4:17677] progress: initialized num users to: 0
[cn4:17677] progress: initialized poll rate to: 23940000
[cn4:17677] progress: event_users_increment setting count to 1
[cn4:17677] mca: base: components_register: registering ess components
[cn4:17677] mca: base: components_register: found loaded component env
[cn4:17677] mca: base: components_register: component env has no register or open function
[cn4:17677] mca: base: components_register: found loaded component hnp
[cn4:17677] mca: base: components_register: component hnp has no register or open function
[cn4:17677] mca: base: components_register: found loaded component pmi
[cn4:17677] mca: base: components_register: component pmi has no register or open function
[cn4:17677] mca: base: components_register: found loaded component singleton
[cn4:17677] mca: base: components_register: component singleton register function successful
[cn4:17677] mca: base: components_register: found loaded component slurm
[cn4:17677] mca: base: components_register: component slurm has no register or open function
[cn4:17677] mca: base: components_register: found loaded component tool
[cn4:17677] mca: base: components_register: component tool has no register or open function
[cn4:17677] mca: base: components_open: opening ess components
[cn4:17677] mca: base: components_open: found loaded component env
[cn4:17677] mca: base: components_open: component env open function successful
[cn4:17677] mca: base: components_open: found loaded component hnp
[cn4:17677] mca: base: components_open: component hnp open function successful
[cn4:17677] mca: base: components_open: found loaded component pmi
[cn4:17677] mca: base: components_open: component pmi open function successful
[cn4:17677] mca: base: components_open: found loaded component singleton
[cn4:17677] mca: base: components_open: component singleton open function successful
[cn4:17677] mca: base: components_open: found loaded component slurm
[cn4:17677] mca: base: components_open: component slurm open function successful
[cn4:17677] mca: base: components_open: found loaded component tool
[cn4:17677] mca: base: components_open: component tool open function successful
[cn4:17677] mca:base:select: Auto-selecting ess components
[cn4:17677] mca:base:select:(  ess) Querying component [env]
[cn4:17677] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn4:17677] mca:base:select:(  ess) Querying component [hnp]
[cn4:17677] mca:base:select:(  ess) Skipping component [hnp]. Query failed to return a module
[cn4:17677] mca:base:select:(  ess) Querying component [pmi]
[cn4:17677] mca:base:select:(  ess) Query of component [pmi] set priority to 35
[cn4:17677] mca:base:select:(  ess) Querying component [singleton]
[cn4:17677] mca:base:select:(  ess) Skipping component [singleton]. Query failed to return a module
[cn4:17677] mca:base:select:(  ess) Querying component [slurm]
[cn4:17677] mca:base:select:(  ess) Query of component [slurm] set priority to 50
[cn4:17677] mca:base:select:(  ess) Querying component [tool]
[cn4:17677] mca:base:select:(  ess) Skipping component [tool]. Query failed to return a module
[cn4:17677] mca:base:select:(  ess) Selected component [slurm]
[cn4:17677] mca: base: close: component env closed
[cn4:17677] mca: base: close: unloading component env
[cn4:17677] mca: base: close: component hnp closed
[cn4:17677] mca: base: close: unloading component hnp
[cn4:17677] mca: base: close: component pmi closed
[cn4:17677] mca: base: close: unloading component pmi
[cn4:17677] mca: base: close: component singleton closed
[cn4:17677] mca: base: close: unloading component singleton
[cn4:17677] mca: base: close: component tool closed
[cn4:17677] mca: base: close: unloading component tool
[cn4:17677] ess:slurm setting name
[cn4:17677] ess:slurm set name to [[15019,0],3]
[cn4:17677] ess:slurm set nodename to cn4
[cn5:14014] ess:slurm setting name
[cn5:14014] ess:slurm set name to [[15019,0],4]
[cn5:14014] ess:slurm set nodename to cn5
[cn5:14014] [[15019,0],4] Topology Info:
[cn5:14014] Type: Machine Number of child objects: 2
	Name=NULL
	total=25100444KB
	DMIProductName=S5520UR
	DMIProductVersion=....................
	DMIBoardVendor="Intel Corporation"
	DMIBoardName=S5520UR
	DMIBoardVersion=E22554-751
	DMIBoardAssetTag=....................
	DMIChassisVendor=...............................
	DMIChassisType=23
	DMIChassisVersion=..................
	DMIChassisAssetTag=....................
	DMIBIOSVendor="Intel Corp."
	DMIBIOSVersion=S5500.86B.01.00.0054.092820101104
	DMIBIOSDate=09/28/2010
	DMISysVendor="Intel Corporation"
	Backend=Linux
	LinuxCgroup=/
	OSName=Linux
	OSRelease=2.6.32-431.3.1.el6.x86_64
	OSVersion="#1 SMP Fri Jan 3 21:39:27 UTC 2014"
	Architecture=x86_64
	Cpuset:  0x000000ff
	Online:  0x000000ff
	Allowed: 0x000000ff
	Bind CPU proc:   TRUE
	Bind CPU thread: TRUE
	Bind MEM proc:   FALSE
	Bind MEM thread: FALSE
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12517532KB
		total=12517532KB
		Cpuset:  0x00000055
		Online:  0x00000055
		Allowed: 0x00000055
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x00000055
			Online:  0x00000055
			Allowed: 0x00000055
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x00000055
				Online:  0x00000055
				Allowed: 0x00000055
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000001
					Online:  0x00000001
					Allowed: 0x00000001
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000001
						Online:  0x00000001
						Allowed: 0x00000001
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000001
							Online:  0x00000001
							Allowed: 0x00000001
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000001
								Online:  0x00000001
								Allowed: 0x00000001
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000004
					Online:  0x00000004
					Allowed: 0x00000004
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000004
						Online:  0x00000004
						Allowed: 0x00000004
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000004
							Online:  0x00000004
							Allowed: 0x00000004
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000004
								Online:  0x00000004
								Allowed: 0x00000004
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000010
					Online:  0x00000010
					Allowed: 0x00000010
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000010
						Online:  0x00000010
						Allowed: 0x00000010
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000010
							Online:  0x00000010
							Allowed: 0x00000010
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000010
								Online:  0x00000010
								Allowed: 0x00000010
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000040
					Online:  0x00000040
					Allowed: 0x00000040
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000040
						Online:  0x00000040
						Allowed: 0x00000040
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000040
							Online:  0x00000040
							Allowed: 0x00000040
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000040
								Online:  0x00000040
								Allowed: 0x00000040
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12582912KB
		total=12582912KB
		Cpuset:  0x000000aa
		Online:  0x000000aa
		Allowed: 0x000000aa
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x000000aa
			Online:  0x000000aa
			Allowed: 0x000000aa
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x000000aa
				Online:  0x000000aa
				Allowed: 0x000000aa
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000002
					Online:  0x00000002
					Allowed: 0x00000002
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000002
						Online:  0x00000002
						Allowed: 0x00000002
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000002
							Online:  0x00000002
							Allowed: 0x00000002
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000002
								Online:  0x00000002
								Allowed: 0x00000002
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000008
					Online:  0x00000008
					Allowed: 0x00000008
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000008
						Online:  0x00000008
						Allowed: 0x00000008
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000008
							Online:  0x00000008
							Allowed: 0x00000008
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000008
								Online:  0x00000008
								Allowed: 0x00000008
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000020
					Online:  0x00000020
					Allowed: 0x00000020
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000020
						Online:  0x00000020
						Allowed: 0x00000020
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000020
							Online:  0x00000020
							Allowed: 0x00000020
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000020
								Online:  0x00000020
								Allowed: 0x00000020
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000080
					Online:  0x00000080
					Allowed: 0x00000080
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000080
						Online:  0x00000080
						Allowed: 0x00000080
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000080
							Online:  0x00000080
							Allowed: 0x00000080
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000080
								Online:  0x00000080
								Allowed: 0x00000080
[cn3:21265] progress: initialized event flag to: 3
[cn3:21265] progress: initialized yield_when_idle to: false
[cn3:21265] progress: initialized num users to: 0
[cn3:21265] progress: initialized poll rate to: 23940000
[cn3:21265] progress: event_users_increment setting count to 1
[cn3:21265] mca: base: components_register: registering ess components
[cn3:21265] mca: base: components_register: found loaded component env
[cn3:21265] mca: base: components_register: component env has no register or open function
[cn3:21265] mca: base: components_register: found loaded component hnp
[cn3:21265] mca: base: components_register: component hnp has no register or open function
[cn3:21265] mca: base: components_register: found loaded component pmi
[cn3:21265] mca: base: components_register: component pmi has no register or open function
[cn3:21265] mca: base: components_register: found loaded component singleton
[cn3:21265] mca: base: components_register: component singleton register function successful
[cn3:21265] mca: base: components_register: found loaded component slurm
[cn3:21265] mca: base: components_register: component slurm has no register or open function
[cn3:21265] mca: base: components_register: found loaded component tool
[cn3:21265] mca: base: components_register: component tool has no register or open function
[cn3:21265] mca: base: components_open: opening ess components
[cn3:21265] mca: base: components_open: found loaded component env
[cn3:21265] mca: base: components_open: component env open function successful
[cn3:21265] mca: base: components_open: found loaded component hnp
[cn3:21265] mca: base: components_open: component hnp open function successful
[cn3:21265] mca: base: components_open: found loaded component pmi
[cn3:21265] mca: base: components_open: component pmi open function successful
[cn3:21265] mca: base: components_open: found loaded component singleton
[cn3:21265] mca: base: components_open: component singleton open function successful
[cn3:21265] mca: base: components_open: found loaded component slurm
[cn3:21265] mca: base: components_open: component slurm open function successful
[cn3:21265] mca: base: components_open: found loaded component tool
[cn3:21265] mca: base: components_open: component tool open function successful
[cn3:21265] mca:base:select: Auto-selecting ess components
[cn3:21265] mca:base:select:(  ess) Querying component [env]
[cn3:21265] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn3:21265] mca:base:select:(  ess) Querying component [hnp]
[cn3:21265] mca:base:select:(  ess) Skipping component [hnp]. Query failed to return a module
[cn3:21265] mca:base:select:(  ess) Querying component [pmi]
[cn3:21265] mca:base:select:(  ess) Query of component [pmi] set priority to 35
[cn3:21265] mca:base:select:(  ess) Querying component [singleton]
[cn3:21265] mca:base:select:(  ess) Skipping component [singleton]. Query failed to return a module
[cn3:21265] mca:base:select:(  ess) Querying component [slurm]
[cn3:21265] mca:base:select:(  ess) Query of component [slurm] set priority to 50
[cn3:21265] mca:base:select:(  ess) Querying component [tool]
[cn3:21265] mca:base:select:(  ess) Skipping component [tool]. Query failed to return a module
[cn3:21265] mca:base:select:(  ess) Selected component [slurm]
[cn3:21265] mca: base: close: component env closed
[cn3:21265] mca: base: close: unloading component env
[cn3:21265] mca: base: close: component hnp closed
[cn3:21265] mca: base: close: unloading component hnp
[cn3:21265] mca: base: close: component pmi closed
[cn3:21265] mca: base: close: unloading component pmi
[cn3:21265] mca: base: close: component singleton closed
[cn3:21265] mca: base: close: unloading component singleton
[cn3:21265] mca: base: close: component tool closed
[cn3:21265] mca: base: close: unloading component tool
[cn2:00814] [[15019,0],1] Topology Info:
[cn2:00814] Type: Machine Number of child objects: 2
	Name=NULL
	total=25100516KB
	DMIProductName=S5520UR
	DMIProductVersion=....................
	DMIBoardVendor="Intel Corporation"
	DMIBoardName=S5520UR
	DMIBoardVersion=E22554-751
	DMIBoardAssetTag=....................
	DMIChassisVendor=...............................
	DMIChassisType=23
	DMIChassisVersion=..................
	DMIChassisAssetTag=....................
	DMIBIOSVendor="Intel Corp."
	DMIBIOSVersion=S5500.86B.01.00.0054.092820101104
	DMIBIOSDate=09/28/2010
	DMISysVendor="Intel Corporation"
	Backend=Linux
	LinuxCgroup=/
	OSName=Linux
	OSRelease=2.6.32-431.3.1.el6.x86_64
	OSVersion="#1 SMP Fri Jan 3 21:39:27 UTC 2014"
	Architecture=x86_64
	Cpuset:  0x000000ff
	Online:  0x000000ff
	Allowed: 0x000000ff
	Bind CPU proc:   TRUE
	Bind CPU thread: TRUE
	Bind MEM proc:   FALSE
	Bind MEM thread: FALSE
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12517604KB
		total=12517604KB
		Cpuset:  0x00000055
		Online:  0x00000055
		Allowed: 0x00000055
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x00000055
			Online:  0x00000055
			Allowed: 0x00000055
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x00000055
				Online:  0x00000055
				Allowed: 0x00000055
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000001
					Online:  0x00000001
					Allowed: 0x00000001
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000001
						Online:  0x00000001
						Allowed: 0x00000001
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000001
							Online:  0x00000001
							Allowed: 0x00000001
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000001
								Online:  0x00000001
								Allowed: 0x00000001
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000004
					Online:  0x00000004
					Allowed: 0x00000004
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000004
						Online:  0x00000004
						Allowed: 0x00000004
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000004
							Online:  0x00000004
							Allowed: 0x00000004
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000004
								Online:  0x00000004
								Allowed: 0x00000004
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000010
					Online:  0x00000010
					Allowed: 0x00000010
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000010
						Online:  0x00000010
						Allowed: 0x00000010
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000010
							Online:  0x00000010
							Allowed: 0x00000010
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000010
								Online:  0x00000010
								Allowed: 0x00000010
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000040
					Online:  0x00000040
					Allowed: 0x00000040
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000040
						Online:  0x00000040
						Allowed: 0x00000040
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000040
							Online:  0x00000040
							Allowed: 0x00000040
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000040
								Online:  0x00000040
								Allowed: 0x00000040
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12582912KB
		total=12582912KB
		Cpuset:  0x000000aa
		Online:  0x000000aa
		Allowed: 0x000000aa
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x000000aa
			Online:  0x000000aa
			Allowed: 0x000000aa
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x000000aa
				Online:  0x000000aa
				Allowed: 0x000000aa
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000002
					Online:  0x00000002
					Allowed: 0x00000002
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000002
						Online:  0x00000002
						Allowed: 0x00000002
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000002
							Online:  0x00000002
							Allowed: 0x00000002
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000002
								Online:  0x00000002
								Allowed: 0x00000002
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000008
					Online:  0x00000008
					Allowed: 0x00000008
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000008
						Online:  0x00000008
						Allowed: 0x00000008
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000008
							Online:  0x00000008
							Allowed: 0x00000008
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000008
								Online:  0x00000008
								Allowed: 0x00000008
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000020
					Online:  0x00000020
					Allowed: 0x00000020
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000020
						Online:  0x00000020
						Allowed: 0x00000020
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000020
							Online:  0x00000020
							Allowed: 0x00000020
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000020
								Online:  0x00000020
								Allowed: 0x00000020
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000080
					Online:  0x00000080
					Allowed: 0x00000080
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000080
						Online:  0x00000080
						Allowed: 0x00000080
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000080
							Online:  0x00000080
							Allowed: 0x00000080
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000080
								Online:  0x00000080
								Allowed: 0x00000080
[cn4:17677] [[15019,0],3] Topology Info:
[cn4:17677] Type: Machine Number of child objects: 2
	Name=NULL
	total=25100516KB
	DMIProductName=S5520UR
	DMIProductVersion=....................
	DMIBoardVendor="Intel Corporation"
	DMIBoardName=S5520UR
	DMIBoardVersion=E22554-751
	DMIBoardAssetTag=....................
	DMIChassisVendor=...............................
	DMIChassisType=23
	DMIChassisVersion=..................
	DMIChassisAssetTag=....................
	DMIBIOSVendor="Intel Corp."
	DMIBIOSVersion=S5500.86B.01.00.0054.092820101104
	DMIBIOSDate=09/28/2010
	DMISysVendor="Intel Corporation"
	Backend=Linux
	LinuxCgroup=/
	OSName=Linux
	OSRelease=2.6.32-431.3.1.el6.x86_64
	OSVersion="#1 SMP Fri Jan 3 21:39:27 UTC 2014"
	Architecture=x86_64
	Cpuset:  0x000000ff
	Online:  0x000000ff
	Allowed: 0x000000ff
	Bind CPU proc:   TRUE
	Bind CPU thread: TRUE
	Bind MEM proc:   FALSE
	Bind MEM thread: FALSE
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12517604KB
		total=12517604KB
		Cpuset:  0x00000055
		Online:  0x00000055
		Allowed: 0x00000055
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x00000055
			Online:  0x00000055
			Allowed: 0x00000055
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x00000055
				Online:  0x00000055
				Allowed: 0x00000055
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000001
					Online:  0x00000001
					Allowed: 0x00000001
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000001
						Online:  0x00000001
						Allowed: 0x00000001
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000001
							Online:  0x00000001
							Allowed: 0x00000001
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000001
								Online:  0x00000001
								Allowed: 0x00000001
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000004
					Online:  0x00000004
					Allowed: 0x00000004
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000004
						Online:  0x00000004
						Allowed: 0x00000004
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000004
							Online:  0x00000004
							Allowed: 0x00000004
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000004
								Online:  0x00000004
								Allowed: 0x00000004
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000010
					Online:  0x00000010
					Allowed: 0x00000010
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000010
						Online:  0x00000010
						Allowed: 0x00000010
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000010
							Online:  0x00000010
							Allowed: 0x00000010
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000010
								Online:  0x00000010
								Allowed: 0x00000010
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000040
					Online:  0x00000040
					Allowed: 0x00000040
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000040
						Online:  0x00000040
						Allowed: 0x00000040
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000040
							Online:  0x00000040
							Allowed: 0x00000040
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000040
								Online:  0x00000040
								Allowed: 0x00000040
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12582912KB
		total=12582912KB
		Cpuset:  0x000000aa
		Online:  0x000000aa
		Allowed: 0x000000aa
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x000000aa
			Online:  0x000000aa
			Allowed: 0x000000aa
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x000000aa
				Online:  0x000000aa
				Allowed: 0x000000aa
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000002
					Online:  0x00000002
					Allowed: 0x00000002
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000002
						Online:  0x00000002
						Allowed: 0x00000002
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000002
							Online:  0x00000002
							Allowed: 0x00000002
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000002
								Online:  0x00000002
								Allowed: 0x00000002
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000008
					Online:  0x00000008
					Allowed: 0x00000008
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000008
						Online:  0x00000008
						Allowed: 0x00000008
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000008
							Online:  0x00000008
							Allowed: 0x00000008
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000008
								Online:  0x00000008
								Allowed: 0x00000008
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000020
					Online:  0x00000020
					Allowed: 0x00000020
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000020
						Online:  0x00000020
						Allowed: 0x00000020
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000020
							Online:  0x00000020
							Allowed: 0x00000020
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000020
								Online:  0x00000020
								Allowed: 0x00000020
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000080
					Online:  0x00000080
					Allowed: 0x00000080
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000080
						Online:  0x00000080
						Allowed: 0x00000080
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000080
							Online:  0x00000080
							Allowed: 0x00000080
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000080
								Online:  0x00000080
								Allowed: 0x00000080
[cn5:14014] mca: base: components_register: registering oob components
[cn5:14014] mca: base: components_register: found loaded component tcp
[cn5:14014] mca: base: components_register: component tcp register function successful
[cn5:14014] mca: base: components_open: opening oob components
[cn5:14014] mca: base: components_open: found loaded component tcp
[cn5:14014] mca: base: components_open: component tcp open function successful
[cn5:14014] mca:oob:select: checking available component tcp
[cn5:14014] mca:oob:select: Querying component [tcp]
[cn5:14014] oob:tcp: component_available called
[cn5:14014] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn5:14014] [[15019,0],4] oob:tcp:init rejecting loopback interface lo
[cn5:14014] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn5:14014] [[15019,0],4] oob:tcp:init creating module for V4 address on interface eth0
[cn5:14014] [[15019,0],4] creating OOB-TCP module for interface eth0
[cn5:14014] [[15019,0],4] oob:tcp:init adding 192.168.1.5 to our list of V4 connections
[cn5:14014] [[15019,0],4] TCP STARTUP
[cn5:14014] [[15019,0],4] attempting to bind to IPv4 port 0
[cn5:14014] [[15019,0],4] assigned IPv4 port 41719
[cn5:14014] mca:oob:select: Adding component to end
[cn5:14014] mca:oob:select: Found 1 active transports
[cn5:14014] mca: base: components_register: registering rml components
[cn5:14014] mca: base: components_register: found loaded component oob
[cn5:14014] mca: base: components_register: component oob has no register or open function
[cn5:14014] mca: base: components_open: opening rml components
[cn5:14014] mca: base: components_open: found loaded component oob
[cn5:14014] mca: base: components_open: component oob open function successful
[cn5:14014] orte_rml_base_select: initializing rml component oob
[cn3:21265] ess:slurm setting name
[cn3:21265] ess:slurm set name to [[15019,0],2]
[cn3:21265] ess:slurm set nodename to cn3
[cn3:21265] [[15019,0],2] Topology Info:
[cn3:21265] Type: Machine Number of child objects: 2
	Name=NULL
	total=25100444KB
	DMIProductName=S5520UR
	DMIProductVersion=....................
	DMIBoardVendor="Intel Corporation"
	DMIBoardName=S5520UR
	DMIBoardVersion=E22554-751
	DMIBoardAssetTag=....................
	DMIChassisVendor=...............................
	DMIChassisType=23
	DMIChassisVersion=..................
	DMIChassisAssetTag=....................
	DMIBIOSVendor="Intel Corp."
	DMIBIOSVersion=S5500.86B.01.00.0054.092820101104
	DMIBIOSDate=09/28/2010
	DMISysVendor="Intel Corporation"
	Backend=Linux
	LinuxCgroup=/
	OSName=Linux
	OSRelease=2.6.32-431.3.1.el6.x86_64
	OSVersion="#1 SMP Fri Jan 3 21:39:27 UTC 2014"
	Architecture=x86_64
	Cpuset:  0x000000ff
	Online:  0x000000ff
	Allowed: 0x000000ff
	Bind CPU proc:   TRUE
	Bind CPU thread: TRUE
	Bind MEM proc:   FALSE
	Bind MEM thread: FALSE
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12517532KB
		total=12517532KB
		Cpuset:  0x00000055
		Online:  0x00000055
		Allowed: 0x00000055
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x00000055
			Online:  0x00000055
			Allowed: 0x00000055
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x00000055
				Online:  0x00000055
				Allowed: 0x00000055
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000001
					Online:  0x00000001
					Allowed: 0x00000001
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000001
						Online:  0x00000001
						Allowed: 0x00000001
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000001
							Online:  0x00000001
							Allowed: 0x00000001
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000001
								Online:  0x00000001
								Allowed: 0x00000001
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000004
					Online:  0x00000004
					Allowed: 0x00000004
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000004
						Online:  0x00000004
						Allowed: 0x00000004
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000004
							Online:  0x00000004
							Allowed: 0x00000004
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000004
								Online:  0x00000004
								Allowed: 0x00000004
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000010
					Online:  0x00000010
					Allowed: 0x00000010
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000010
						Online:  0x00000010
						Allowed: 0x00000010
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000010
							Online:  0x00000010
							Allowed: 0x00000010
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000010
								Online:  0x00000010
								Allowed: 0x00000010
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000040
					Online:  0x00000040
					Allowed: 0x00000040
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000040
						Online:  0x00000040
						Allowed: 0x00000040
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000040
							Online:  0x00000040
							Allowed: 0x00000040
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000040
								Online:  0x00000040
								Allowed: 0x00000040
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12582912KB
		total=12582912KB
		Cpuset:  0x000000aa
		Online:  0x000000aa
		Allowed: 0x000000aa
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x000000aa
			Online:  0x000000aa
			Allowed: 0x000000aa
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x000000aa
				Online:  0x000000aa
				Allowed: 0x000000aa
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000002
					Online:  0x00000002
					Allowed: 0x00000002
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000002
						Online:  0x00000002
						Allowed: 0x00000002
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000002
							Online:  0x00000002
							Allowed: 0x00000002
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000002
								Online:  0x00000002
								Allowed: 0x00000002
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000008
					Online:  0x00000008
					Allowed: 0x00000008
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000008
						Online:  0x00000008
						Allowed: 0x00000008
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000008
							Online:  0x00000008
							Allowed: 0x00000008
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000008
								Online:  0x00000008
								Allowed: 0x00000008
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000020
					Online:  0x00000020
					Allowed: 0x00000020
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000020
						Online:  0x00000020
						Allowed: 0x00000020
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000020
							Online:  0x00000020
							Allowed: 0x00000020
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000020
								Online:  0x00000020
								Allowed: 0x00000020
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000080
					Online:  0x00000080
					Allowed: 0x00000080
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000080
						Online:  0x00000080
						Allowed: 0x00000080
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000080
							Online:  0x00000080
							Allowed: 0x00000080
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000080
								Online:  0x00000080
								Allowed: 0x00000080
[cn5:14014] mca: base: components_register: registering routed components
[cn5:14014] mca: base: components_register: found loaded component binomial
[cn5:14014] mca: base: components_register: component binomial has no register or open function
[cn5:14014] mca: base: components_register: found loaded component debruijn
[cn5:14014] mca: base: components_register: component debruijn has no register or open function
[cn5:14014] mca: base: components_register: found loaded component direct
[cn5:14014] mca: base: components_register: component direct has no register or open function
[cn5:14014] mca: base: components_register: found loaded component radix
[cn5:14014] mca: base: components_register: component radix register function successful
[cn5:14014] mca: base: components_open: opening routed components
[cn5:14014] mca: base: components_open: found loaded component binomial
[cn5:14014] mca: base: components_open: found loaded component debruijn
[cn5:14014] mca: base: components_open: found loaded component direct
[cn5:14014] mca: base: components_open: found loaded component radix
[cn5:14014] mca:base:select: Auto-selecting routed components
[cn5:14014] mca:base:select:(routed) Querying component [binomial]
[cn5:14014] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn5:14014] mca:base:select:(routed) Querying component [debruijn]
[cn5:14014] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn5:14014] mca:base:select:(routed) Querying component [direct]
[cn5:14014] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn5:14014] mca:base:select:(routed) Querying component [radix]
[cn5:14014] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn5:14014] mca:base:select:(routed) Selected component [binomial]
[cn5:14014] mca: base: close: unloading component debruijn
[cn5:14014] mca: base: close: unloading component direct
[cn5:14014] mca: base: close: unloading component radix
[cn5:14014] orte_routed_base_select: initializing selected component binomial
[cn2:00814] mca: base: components_register: registering oob components
[cn2:00814] mca: base: components_register: found loaded component tcp
[cn2:00814] mca: base: components_register: component tcp register function successful
[cn2:00814] mca: base: components_open: opening oob components
[cn2:00814] mca: base: components_open: found loaded component tcp
[cn2:00814] mca: base: components_open: component tcp open function successful
[cn5:14014] mca: base: components_register: registering dstore components
[cn5:14014] mca: base: components_register: found loaded component hash
[cn5:14014] mca: base: components_register: component hash register function successful
[cn5:14014] mca: base: components_open: opening dstore components
[cn5:14014] mca: base: components_open: found loaded component hash
[cn5:14014] mca:dstore:select: checking available component hash
[cn5:14014] mca:dstore:select: Querying component [hash]
[cn2:00814] mca:oob:select: checking available component tcp
[cn2:00814] mca:oob:select: Querying component [tcp]
[cn2:00814] oob:tcp: component_available called
[cn5:14014] mca: base: components_register: registering grpcomm components
[cn5:14014] mca: base: components_register: found loaded component bad
[cn5:14014] mca: base: components_register: component bad has no register or open function
[cn5:14014] mca: base: components_register: found loaded component pmi
[cn5:14014] mca: base: components_register: component pmi register function successful
[cn2:00814] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn2:00814] [[15019,0],1] oob:tcp:init rejecting loopback interface lo
[cn2:00814] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn2:00814] [[15019,0],1] oob:tcp:init creating module for V4 address on interface eth0
[cn2:00814] [[15019,0],1] creating OOB-TCP module for interface eth0
[cn2:00814] [[15019,0],1] oob:tcp:init adding 192.168.1.2 to our list of V4 connections
[cn2:00814] [[15019,0],1] TCP STARTUP
[cn2:00814] [[15019,0],1] attempting to bind to IPv4 port 0
[cn2:00814] [[15019,0],1] assigned IPv4 port 49722
[cn2:00814] mca:oob:select: Adding component to end
[cn2:00814] mca:oob:select: Found 1 active transports
[cn5:14014] mca: base: components_open: opening grpcomm components
[cn5:14014] mca: base: components_open: found loaded component bad
[cn5:14014] mca: base: components_open: component bad open function successful
[cn5:14014] mca: base: components_open: found loaded component pmi
[cn5:14014] mca: base: components_open: component pmi open function successful
[cn5:14014] mca:base:select: Auto-selecting grpcomm components
[cn5:14014] mca:base:select:(grpcomm) Querying component [bad]
[cn5:14014] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn5:14014] mca:base:select:(grpcomm) Querying component [pmi]
[cn5:14014] mca:base:select:(grpcomm) Skipping component [pmi]. Query failed to return a module
[cn5:14014] mca:base:select:(grpcomm) Selected component [bad]
[cn5:14014] mca: base: close: component pmi closed
[cn5:14014] mca: base: close: unloading component pmi
[cn2:00814] mca: base: components_register: registering rml components
[cn2:00814] mca: base: components_register: found loaded component oob
[cn5:14014] [[15019,0],4] grpcomm:base:receive start comm
[cn5:14014] [[15019,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn5:14014] [[15019,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 15
[cn5:14014] [[15019,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 32
[cn5:14014] [[15019,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 11
[cn2:00814] mca: base: components_register: component oob has no register or open function
[cn2:00814] mca: base: components_open: opening rml components
[cn2:00814] mca: base: components_open: found loaded component oob
[cn2:00814] mca: base: components_open: component oob open function successful
[cn2:00814] orte_rml_base_select: initializing rml component oob
[cn5:14014] [[15019,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn5:14014] [[15019,0],4] routed:binomial rank 0 parent 0 me 4 num_procs 5
[cn5:14014] [[15019,0],4] routed:binomial find children of rank 0
[cn5:14014] [[15019,0],4] routed:binomial find children checking peer 1
[cn5:14014] [[15019,0],4] routed:binomial find children computing tree
[cn5:14014] [[15019,0],4] routed:binomial rank 1 parent 0 me 4 num_procs 5
[cn5:14014] [[15019,0],4] routed:binomial find children of rank 1
[cn5:14014] [[15019,0],4] routed:binomial find children checking peer 3
[cn5:14014] [[15019,0],4] routed:binomial find children computing tree
[cn5:14014] [[15019,0],4] routed:binomial rank 3 parent 1 me 4 num_procs 5
[cn5:14014] [[15019,0],4] routed:binomial find children of rank 3
[cn5:14014] [[15019,0],4] routed:binomial find children checking peer 7
[cn5:14014] [[15019,0],4] routed:binomial find children checking peer 5
[cn5:14014] [[15019,0],4] routed:binomial find children checking peer 2
[cn5:14014] [[15019,0],4] routed:binomial find children computing tree
[cn5:14014] [[15019,0],4] routed:binomial rank 2 parent 0 me 4 num_procs 5
[cn5:14014] [[15019,0],4] routed:binomial find children of rank 2
[cn5:14014] [[15019,0],4] routed:binomial find children checking peer 6
[cn5:14014] [[15019,0],4] routed:binomial find children checking peer 4
[cn5:14014] [[15019,0],4] routed:binomial find children computing tree
[cn5:14014] [[15019,0],4] routed:binomial rank 4 parent 0 me 4 num_procs 5
[cn5:14014] [[15019,0],4] routed:binomial find children returning found value 0
[cn5:14014] [[15019,0],4]: parent 0 num_children 0
[cn5:14014] [[15019,0],4] setting up session dir with
	tmpdir: UNDEF
	host cn5
[cn2:00814] mca: base: components_register: registering routed components
[cn2:00814] mca: base: components_register: found loaded component binomial
[cn2:00814] mca: base: components_register: component binomial has no register or open function
[cn2:00814] mca: base: components_register: found loaded component debruijn
[cn2:00814] mca: base: components_register: component debruijn has no register or open function
[cn2:00814] mca: base: components_register: found loaded component direct
[cn2:00814] mca: base: components_register: component direct has no register or open function
[cn2:00814] mca: base: components_register: found loaded component radix
[cn2:00814] mca: base: components_register: component radix register function successful
[cn2:00814] mca: base: components_open: opening routed components
[cn2:00814] mca: base: components_open: found loaded component binomial
[cn2:00814] mca: base: components_open: found loaded component debruijn
[cn2:00814] mca: base: components_open: found loaded component direct
[cn2:00814] mca: base: components_open: found loaded component radix
[cn2:00814] mca:base:select: Auto-selecting routed components
[cn2:00814] mca:base:select:(routed) Querying component [binomial]
[cn2:00814] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn2:00814] mca:base:select:(routed) Querying component [debruijn]
[cn2:00814] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn2:00814] mca:base:select:(routed) Querying component [direct]
[cn2:00814] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn2:00814] mca:base:select:(routed) Querying component [radix]
[cn2:00814] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn2:00814] mca:base:select:(routed) Selected component [binomial]
[cn2:00814] mca: base: close: unloading component debruijn
[cn2:00814] mca: base: close: unloading component direct
[cn2:00814] mca: base: close: unloading component radix
[cn2:00814] orte_routed_base_select: initializing selected component binomial
[cn4:17677] mca: base: components_register: registering oob components
[cn4:17677] mca: base: components_register: found loaded component tcp
[cn4:17677] mca: base: components_register: component tcp register function successful
[cn4:17677] mca: base: components_open: opening oob components
[cn4:17677] mca: base: components_open: found loaded component tcp
[cn4:17677] mca: base: components_open: component tcp open function successful
[cn4:17677] mca:oob:select: checking available component tcp
[cn4:17677] mca:oob:select: Querying component [tcp]
[cn4:17677] oob:tcp: component_available called
[cn4:17677] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn4:17677] [[15019,0],3] oob:tcp:init rejecting loopback interface lo
[cn4:17677] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn4:17677] [[15019,0],3] oob:tcp:init creating module for V4 address on interface eth0
[cn4:17677] [[15019,0],3] creating OOB-TCP module for interface eth0
[cn4:17677] [[15019,0],3] oob:tcp:init adding 192.168.1.4 to our list of V4 connections
[cn4:17677] [[15019,0],3] TCP STARTUP
[cn4:17677] [[15019,0],3] attempting to bind to IPv4 port 0
[cn4:17677] [[15019,0],3] assigned IPv4 port 43236
[cn4:17677] mca:oob:select: Adding component to end
[cn4:17677] mca:oob:select: Found 1 active transports
[cn4:17677] mca: base: components_register: registering rml components
[cn4:17677] mca: base: components_register: found loaded component oob
[cn4:17677] mca: base: components_register: component oob has no register or open function
[cn4:17677] mca: base: components_open: opening rml components
[cn4:17677] mca: base: components_open: found loaded component oob
[cn4:17677] mca: base: components_open: component oob open function successful
[cn4:17677] orte_rml_base_select: initializing rml component oob
[cn2:00814] mca: base: components_register: registering dstore components
[cn2:00814] mca: base: components_register: found loaded component hash
[cn2:00814] mca: base: components_register: component hash register function successful
[cn2:00814] mca: base: components_open: opening dstore components
[cn2:00814] mca: base: components_open: found loaded component hash
[cn2:00814] mca:dstore:select: checking available component hash
[cn2:00814] mca:dstore:select: Querying component [hash]
[cn2:00814] mca: base: components_register: registering grpcomm components
[cn2:00814] mca: base: components_register: found loaded component bad
[cn2:00814] mca: base: components_register: component bad has no register or open function
[cn2:00814] mca: base: components_register: found loaded component pmi
[cn2:00814] mca: base: components_register: component pmi register function successful
[cn2:00814] mca: base: components_open: opening grpcomm components
[cn2:00814] mca: base: components_open: found loaded component bad
[cn2:00814] mca: base: components_open: component bad open function successful
[cn2:00814] mca: base: components_open: found loaded component pmi
[cn2:00814] mca: base: components_open: component pmi open function successful
[cn2:00814] mca:base:select: Auto-selecting grpcomm components
[cn2:00814] mca:base:select:(grpcomm) Querying component [bad]
[cn2:00814] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn2:00814] mca:base:select:(grpcomm) Querying component [pmi]
[cn2:00814] mca:base:select:(grpcomm) Skipping component [pmi]. Query failed to return a module
[cn2:00814] mca:base:select:(grpcomm) Selected component [bad]
[cn2:00814] mca: base: close: component pmi closed
[cn2:00814] mca: base: close: unloading component pmi
[cn2:00814] [[15019,0],1] grpcomm:base:receive start comm
[cn2:00814] [[15019,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn2:00814] [[15019,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 15
[cn2:00814] [[15019,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 32
[cn2:00814] [[15019,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 11
[cn2:00814] [[15019,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn2:00814] [[15019,0],1] routed:binomial rank 0 parent 0 me 1 num_procs 5
[cn2:00814] [[15019,0],1] routed:binomial find children of rank 0
[cn2:00814] [[15019,0],1] routed:binomial find children checking peer 1
[cn2:00814] [[15019,0],1] routed:binomial find children computing tree
[cn2:00814] [[15019,0],1] routed:binomial rank 1 parent 0 me 1 num_procs 5
[cn2:00814] [[15019,0],1] routed:binomial 1 found child 3
[cn2:00814] [[15019,0],1] routed:binomial rank 0 parent 0 me 3 num_procs 5
[cn2:00814] [[15019,0],1] routed:binomial find children of rank 0
[cn2:00814] [[15019,0],1] routed:binomial find children checking peer 1
[cn2:00814] [[15019,0],1] routed:binomial find children computing tree
[cn2:00814] [[15019,0],1] routed:binomial rank 1 parent 0 me 3 num_procs 5
[cn2:00814] [[15019,0],1] routed:binomial find children of rank 1
[cn2:00814] [[15019,0],1] routed:binomial find children checking peer 3
[cn2:00814] [[15019,0],1] routed:binomial find children computing tree
[cn2:00814] [[15019,0],1] routed:binomial rank 3 parent 1 me 3 num_procs 5
[cn2:00814] [[15019,0],1] routed:binomial find children returning found value 1
[cn2:00814] [[15019,0],1] routed:binomial find children returning found value 1
[cn2:00814] [[15019,0],1] routed:binomial find children returning found value 0
[cn2:00814] [[15019,0],1]: parent 0 num_children 1
[cn2:00814] [[15019,0],1]: 	child 3
[cn2:00814] [[15019,0],1] setting up session dir with
	tmpdir: UNDEF
	host cn2
[cn4:17677] mca: base: components_register: registering routed components
[cn4:17677] mca: base: components_register: found loaded component binomial
[cn4:17677] mca: base: components_register: component binomial has no register or open function
[cn4:17677] mca: base: components_register: found loaded component debruijn
[cn4:17677] mca: base: components_register: component debruijn has no register or open function
[cn4:17677] mca: base: components_register: found loaded component direct
[cn4:17677] mca: base: components_register: component direct has no register or open function
[cn4:17677] mca: base: components_register: found loaded component radix
[cn4:17677] mca: base: components_register: component radix register function successful
[cn4:17677] mca: base: components_open: opening routed components
[cn4:17677] mca: base: components_open: found loaded component binomial
[cn4:17677] mca: base: components_open: found loaded component debruijn
[cn4:17677] mca: base: components_open: found loaded component direct
[cn4:17677] mca: base: components_open: found loaded component radix
[cn4:17677] mca:base:select: Auto-selecting routed components
[cn4:17677] mca:base:select:(routed) Querying component [binomial]
[cn4:17677] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn4:17677] mca:base:select:(routed) Querying component [debruijn]
[cn4:17677] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn4:17677] mca:base:select:(routed) Querying component [direct]
[cn4:17677] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn4:17677] mca:base:select:(routed) Querying component [radix]
[cn4:17677] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn4:17677] mca:base:select:(routed) Selected component [binomial]
[cn4:17677] mca: base: close: unloading component debruijn
[cn4:17677] mca: base: close: unloading component direct
[cn4:17677] mca: base: close: unloading component radix
[cn4:17677] orte_routed_base_select: initializing selected component binomial
[cn5:14014] procdir: /tmp/openmpi-sessions-artpol@cn5_0/15019/0/4
[cn5:14014] jobdir: /tmp/openmpi-sessions-artpol@cn5_0/15019/0
[cn5:14014] top: openmpi-sessions-artpol@cn5_0
[cn5:14014] tmp: /tmp
[cn5:14014] sess_dir_cleanup: job session dir does not exist
[cn5:14014] procdir: /tmp/openmpi-sessions-artpol@cn5_0/15019/0/4
[cn5:14014] jobdir: /tmp/openmpi-sessions-artpol@cn5_0/15019/0
[cn5:14014] top: openmpi-sessions-artpol@cn5_0
[cn5:14014] tmp: /tmp
[cn5:14014] [[15019,0],4] routed_binomial: init routes for daemon job [15019,0]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn5:14014] [[15019,0],4] routed_binomial: completed init routes
[cn5:14014] [[15019,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 3
[cn5:14014] [[15019,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 21
[cn5:14014] [[15019,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 45
[cn5:14014] [[15019,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 46
[cn5:14014] [[15019,0],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 1
Daemon [[15019,0],4] checking in as pid 14014 on host cn5
[cn5:14014] progress: progress_set_yield_when_idle to false
[cn5:14014] progress: set_event_flag setting to 1
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 10
[cn5:14014] [[15019,0],4] orted: up and running - waiting for commands!
[cn5:14014] [[15019,0],4] posting recv
[cn5:14014] [[15019,0],4] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn5:14014] [[15019,0],4] posting recv
[cn5:14014] [[15019,0],4] posting persistent recv on tag 15 for peer [[WILDCARD],WILDCARD]
[cn5:14014] [[15019,0],4] posting recv
[cn5:14014] [[15019,0],4] posting persistent recv on tag 32 for peer [[WILDCARD],WILDCARD]
[cn5:14014] [[15019,0],4] posting recv
[cn5:14014] [[15019,0],4] posting persistent recv on tag 11 for peer [[WILDCARD],WILDCARD]
[cn5:14014] [[15019,0],4] posting recv
[cn5:14014] [[15019,0],4] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn5:14014] [[15019,0],4]: set_addr to uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn5:14014] [[15019,0],4]:set_addr checking if peer [[15019,0],0] is reachable via component tcp
[cn5:14014] [[15019,0],4] oob:tcp: working peer [[15019,0],0] address tcp://192.168.1.1,192.168.10.2:57056
[cn3:21265] mca: base: components_register: registering oob components
[cn3:21265] mca: base: components_register: found loaded component tcp
[cn5:14014] [[15019,0],4] PEER [[15019,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:14014] [[15019,0],4] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn5:14014] [[15019,0],4]:tcp set addr for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn1:14018] [[15019,0],0] mca_oob_tcp_listen_thread: new connection: (14, 0) 192.168.1.5:33190
[cn5:14014] [[15019,0],4]: peer [[15019,0],0] is reachable via component tcp
[cn5:14014] [[15019,0],4] posting recv
[cn5:14014] [[15019,0],4] posting persistent recv on tag 3 for peer [[WILDCARD],WILDCARD]
[cn5:14014] [[15019,0],4] posting recv
[cn5:14014] [[15019,0],4] posting persistent recv on tag 21 for peer [[WILDCARD],WILDCARD]
[cn5:14014] [[15019,0],4] posting recv
[cn5:14014] [[15019,0],4] posting persistent recv on tag 45 for peer [[WILDCARD],WILDCARD]
[cn5:14014] [[15019,0],4] posting recv
[cn5:14014] [[15019,0],4] posting persistent recv on tag 46 for peer [[WILDCARD],WILDCARD]
[cn5:14014] [[15019,0],4] posting recv
[cn1:14018] [[15019,0],0] connection_handler: working connection (14, 2) 192.168.1.5:33190
[cn5:14014] [[15019,0],4] posting persistent recv on tag 1 for peer [[WILDCARD],WILDCARD]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 10
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4]:tcp:processing set_peer cmd for interface eth0
[cn5:14014] [[15019,0],4] SET_PEER ADDING PEER [[15019,0],0]
[cn1:14018] [[15019,0],0] accept_connection: 192.168.1.5:33190
[cn5:14014] [[15019,0],4] set_peer: peer [[15019,0],0] is listening on interface eth0 net 192.168.1.1 port 57056
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:10
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:10
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called
[cn5:14014] [[15019,0],4] tcp:send_nb: initiating connection to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],0]
[cn5:14014] [[15019,0],4] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] via interface eth0
[cn1:14018] [[15019,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 14
[cn5:14014] [[15019,0],4] oob:tcp:peer creating socket to [[15019,0],0]
[cn1:14018] [[15019,0],0] waiting for connect ack from UNKNOWN
[cn5:14014] [[15019,0],4] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] via interface eth0 on socket 9
[cn1:14018] [[15019,0],0] connect ack received from UNKNOWN
[cn1:14018] [[15019,0],0] connect-ack recvd from UNKNOWN
[cn5:14014] [[15019,0],4] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] on 192.168.1.1:57056 - 0 retries
[cn1:14018] [[15019,0],0] mca_oob_tcp_recv_connect: connection from new peer
[cn5:14014] [[15019,0],4] waiting for connect completion to [[15019,0],0] - activating send event
[cn1:14018] [[15019,0],0] connect-ack header from [[15019,0],4] is okay
[cn1:14018] [[15019,0],0] waiting for connect ack from [[15019,0],4]
[cn1:14018] [[15019,0],0] connect ack received from [[15019,0],4]
[cn1:14018] [[15019,0],0] connect-ack version from [[15019,0],4] matches ours
[cn1:14018] [[15019,0],0] connect-ack [[15019,0],4] authenticated
[cn1:14018] [[15019,0],0] tcp:peer_accept called for peer [[15019,0],4] in state UNKNOWN on socket 14
[cn1:14018] [[15019,0],0] SEND CONNECT ACK
[cn1:14018] fetching data from INTERNAL dstore
[cn1:14018] dstore:hash:fetch: searching for key opal.cred on proc 984285184
[cn1:14018] dstore_hash:fetch key opal.cred for proc 984285184 not found
[cn1:14018] [[15019,0],0] send blocking of 47 bytes to socket 14
[cn1:14018] [[15019,0],0] blocking send complete to socket 14
[cn1:14018] [[15019,0],0]-[[15019,0],4] tcp_peer_connected on socket 14
[cn1:14018] [[15019,0],0] routed_binomial_update: [[15019,0],4] --> [[15019,0],4]
[cn1:14018] [[15019,0],0]-[[15019,0],4] accepted: 192.168.1.1 - 192.168.1.5 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:14018] [[15019,0],0] tcp:set_module called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler CONNECTING
[cn5:14014] [[15019,0],4]:tcp:complete_connect called for peer [[15019,0],0] on socket 9
[cn5:14014] [[15019,0],4] tcp_peer_complete_connect: sending ack to [[15019,0],0]
[cn5:14014] [[15019,0],4] SEND CONNECT ACK
[cn5:14014] fetching data from INTERNAL dstore
[cn5:14014] dstore:hash:fetch: searching for key opal.cred on proc 18164154368
[cn5:14014] dstore_hash:fetch key opal.cred for proc 18164154368 not found
[cn5:14014] [[15019,0],4] send blocking of 47 bytes to socket 9
[cn5:14014] [[15019,0],4] blocking send complete to socket 9
[cn5:14014] [[15019,0],4] tcp_peer_complete_connect: setting read event on connection to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:tcp:recv:handler called for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] RECV CONNECT ACK FROM [[15019,0],0] ON SOCKET 9
[cn5:14014] [[15019,0],4] waiting for connect ack from [[15019,0],0]
[cn5:14014] [[15019,0],4] connect ack received from [[15019,0],0]
[cn5:14014] [[15019,0],4] connect-ack recvd from [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] connect-ack header from [[15019,0],0] is okay
[cn5:14014] [[15019,0],4] waiting for connect ack from [[15019,0],0]
[cn5:14014] [[15019,0],4] connect ack received from [[15019,0],0]
[cn5:14014] [[15019,0],4] connect-ack version from [[15019,0],0] matches ours
[cn5:14014] [[15019,0],4] connect-ack [[15019,0],0] authenticated
[cn5:14014] [[15019,0],4]-[[15019,0],0] tcp_peer_connected on socket 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 96
[cn5:14014] [[15019,0],4] routed_binomial_update: [[15019,0],0] --> [[15019,0],0]
[cn5:14014] [[15019,0],4]-[[15019,0],0] connected: 192.168.1.5 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn5:14014] [[15019,0],4]:tcp:recv:handler starting send/recv events
[cn5:14014] [[15019,0],4] tcp:set_module called for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 96 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] mca: base: components_register: component tcp register function successful
[cn3:21265] mca: base: components_open: opening oob components
[cn3:21265] mca: base: components_open: found loaded component tcp
[cn3:21265] mca: base: components_open: component tcp open function successful
[cn3:21265] mca:oob:select: checking available component tcp
[cn3:21265] mca:oob:select: Querying component [tcp]
[cn3:21265] oob:tcp: component_available called
[cn3:21265] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn3:21265] [[15019,0],2] oob:tcp:init rejecting loopback interface lo
[cn3:21265] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn3:21265] [[15019,0],2] oob:tcp:init creating module for V4 address on interface eth0
[cn3:21265] [[15019,0],2] creating OOB-TCP module for interface eth0
[cn3:21265] [[15019,0],2] oob:tcp:init adding 192.168.1.3 to our list of V4 connections
[cn3:21265] [[15019,0],2] TCP STARTUP
[cn3:21265] [[15019,0],2] attempting to bind to IPv4 port 0
[cn3:21265] [[15019,0],2] assigned IPv4 port 38382
[cn3:21265] mca:oob:select: Adding component to end
[cn3:21265] mca:oob:select: Found 1 active transports
[cn3:21265] mca: base: components_register: registering rml components
[cn3:21265] mca: base: components_register: found loaded component oob
[cn3:21265] mca: base: components_register: component oob has no register or open function
[cn3:21265] mca: base: components_open: opening rml components
[cn3:21265] mca: base: components_open: found loaded component oob
[cn3:21265] mca: base: components_open: component oob open function successful
[cn3:21265] orte_rml_base_select: initializing rml component oob
[cn4:17677] mca: base: components_register: registering dstore components
[cn4:17677] mca: base: components_register: found loaded component hash
[cn4:17677] mca: base: components_register: component hash register function successful
[cn4:17677] mca: base: components_open: opening dstore components
[cn4:17677] mca: base: components_open: found loaded component hash
[cn4:17677] mca:dstore:select: checking available component hash
[cn4:17677] mca:dstore:select: Querying component [hash]
[cn4:17677] mca: base: components_register: registering grpcomm components
[cn4:17677] mca: base: components_register: found loaded component bad
[cn4:17677] mca: base: components_register: component bad has no register or open function
[cn4:17677] mca: base: components_register: found loaded component pmi
[cn4:17677] mca: base: components_register: component pmi register function successful
[cn4:17677] mca: base: components_open: opening grpcomm components
[cn4:17677] mca: base: components_open: found loaded component bad
[cn4:17677] mca: base: components_open: component bad open function successful
[cn4:17677] mca: base: components_open: found loaded component pmi
[cn4:17677] mca: base: components_open: component pmi open function successful
[cn4:17677] mca:base:select: Auto-selecting grpcomm components
[cn4:17677] mca:base:select:(grpcomm) Querying component [bad]
[cn4:17677] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn4:17677] mca:base:select:(grpcomm) Querying component [pmi]
[cn4:17677] mca:base:select:(grpcomm) Skipping component [pmi]. Query failed to return a module
[cn4:17677] mca:base:select:(grpcomm) Selected component [bad]
[cn4:17677] mca: base: close: component pmi closed
[cn4:17677] mca: base: close: unloading component pmi
[cn4:17677] [[15019,0],3] grpcomm:base:receive start comm
[cn4:17677] [[15019,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn4:17677] [[15019,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 15
[cn4:17677] [[15019,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 32
[cn4:17677] [[15019,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 11
[cn3:21265] mca: base: components_register: registering routed components
[cn3:21265] mca: base: components_register: found loaded component binomial
[cn3:21265] mca: base: components_register: component binomial has no register or open function
[cn3:21265] mca: base: components_register: found loaded component debruijn
[cn3:21265] mca: base: components_register: component debruijn has no register or open function
[cn3:21265] mca: base: components_register: found loaded component direct
[cn3:21265] mca: base: components_register: component direct has no register or open function
[cn3:21265] mca: base: components_register: found loaded component radix
[cn3:21265] mca: base: components_register: component radix register function successful
[cn3:21265] mca: base: components_open: opening routed components
[cn3:21265] mca: base: components_open: found loaded component binomial
[cn3:21265] mca: base: components_open: found loaded component debruijn
[cn3:21265] mca: base: components_open: found loaded component direct
[cn3:21265] mca: base: components_open: found loaded component radix
[cn3:21265] mca:base:select: Auto-selecting routed components
[cn3:21265] mca:base:select:(routed) Querying component [binomial]
[cn3:21265] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn3:21265] mca:base:select:(routed) Querying component [debruijn]
[cn3:21265] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn3:21265] mca:base:select:(routed) Querying component [direct]
[cn3:21265] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn3:21265] mca:base:select:(routed) Querying component [radix]
[cn3:21265] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn3:21265] mca:base:select:(routed) Selected component [binomial]
[cn3:21265] mca: base: close: unloading component debruijn
[cn3:21265] mca: base: close: unloading component direct
[cn3:21265] mca: base: close: unloading component radix
[cn3:21265] orte_routed_base_select: initializing selected component binomial
[cn4:17677] [[15019,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn4:17677] [[15019,0],3] routed:binomial rank 0 parent 0 me 3 num_procs 5
[cn4:17677] [[15019,0],3] routed:binomial find children of rank 0
[cn4:17677] [[15019,0],3] routed:binomial find children checking peer 1
[cn4:17677] [[15019,0],3] routed:binomial find children computing tree
[cn4:17677] [[15019,0],3] routed:binomial rank 1 parent 0 me 3 num_procs 5
[cn4:17677] [[15019,0],3] routed:binomial find children of rank 1
[cn4:17677] [[15019,0],3] routed:binomial find children checking peer 3
[cn4:17677] [[15019,0],3] routed:binomial find children computing tree
[cn4:17677] [[15019,0],3] routed:binomial rank 3 parent 1 me 3 num_procs 5
[cn4:17677] [[15019,0],3] routed:binomial find children returning found value 1
[cn4:17677] [[15019,0],3] routed:binomial find children returning found value 1
[cn4:17677] [[15019,0],3]: parent 1 num_children 0
[cn4:17677] [[15019,0],3] setting up session dir with
	tmpdir: UNDEF
	host cn4
[cn4:17677] procdir: /tmp/openmpi-sessions-artpol@cn4_0/15019/0/3
[cn4:17677] jobdir: /tmp/openmpi-sessions-artpol@cn4_0/15019/0
[cn4:17677] top: openmpi-sessions-artpol@cn4_0
[cn4:17677] tmp: /tmp
[cn4:17677] sess_dir_cleanup: job session dir does not exist
[cn4:17677] procdir: /tmp/openmpi-sessions-artpol@cn4_0/15019/0/3
[cn4:17677] jobdir: /tmp/openmpi-sessions-artpol@cn4_0/15019/0
[cn4:17677] top: openmpi-sessions-artpol@cn4_0
[cn4:17677] tmp: /tmp
[cn4:17677] [[15019,0],3] routed_binomial: init routes for daemon job [15019,0]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn4:17677] [[15019,0],3] routed_binomial: completed init routes
[cn2:00814] procdir: /tmp/openmpi-sessions-artpol@cn2_0/15019/0/1
[cn2:00814] jobdir: /tmp/openmpi-sessions-artpol@cn2_0/15019/0
[cn2:00814] top: openmpi-sessions-artpol@cn2_0
[cn2:00814] tmp: /tmp
[cn2:00814] sess_dir_cleanup: job session dir does not exist
[cn2:00814] procdir: /tmp/openmpi-sessions-artpol@cn2_0/15019/0/1
[cn2:00814] jobdir: /tmp/openmpi-sessions-artpol@cn2_0/15019/0
[cn2:00814] top: openmpi-sessions-artpol@cn2_0
[cn2:00814] tmp: /tmp
[cn2:00814] [[15019,0],1] routed_binomial: init routes for daemon job [15019,0]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn2:00814] [[15019,0],1] routed_binomial: completed init routes
[cn2:00814] [[15019,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 3
[cn2:00814] [[15019,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 21
[cn2:00814] [[15019,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 45
[cn2:00814] [[15019,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 46
[cn2:00814] [[15019,0],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 1
Daemon [[15019,0],1] checking in as pid 814 on host cn2
[cn2:00814] progress: progress_set_yield_when_idle to false
[cn2:00814] progress: set_event_flag setting to 1
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 10
[cn2:00814] [[15019,0],1] orted: up and running - waiting for commands!
[cn2:00814] [[15019,0],1] posting recv
[cn2:00814] [[15019,0],1] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn2:00814] [[15019,0],1] posting recv
[cn2:00814] [[15019,0],1] posting persistent recv on tag 15 for peer [[WILDCARD],WILDCARD]
[cn2:00814] [[15019,0],1] posting recv
[cn2:00814] [[15019,0],1] posting persistent recv on tag 32 for peer [[WILDCARD],WILDCARD]
[cn2:00814] [[15019,0],1] posting recv
[cn2:00814] [[15019,0],1] posting persistent recv on tag 11 for peer [[WILDCARD],WILDCARD]
[cn2:00814] [[15019,0],1] posting recv
[cn2:00814] [[15019,0],1] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn2:00814] [[15019,0],1]: set_addr to uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn2:00814] [[15019,0],1]:set_addr checking if peer [[15019,0],0] is reachable via component tcp
[cn2:00814] [[15019,0],1] oob:tcp: working peer [[15019,0],0] address tcp://192.168.1.1,192.168.10.2:57056
[cn2:00814] [[15019,0],1] PEER [[15019,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn2:00814] [[15019,0],1] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00814] [[15019,0],1]:tcp set addr for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn1:14018] [[15019,0],0] mca_oob_tcp_listen_thread: new connection: (15, 11) 192.168.1.2:44866
[cn2:00814] [[15019,0],1]: peer [[15019,0],0] is reachable via component tcp
[cn2:00814] [[15019,0],1] posting recv
[cn2:00814] [[15019,0],1] posting persistent recv on tag 3 for peer [[WILDCARD],WILDCARD]
[cn2:00814] [[15019,0],1] posting recv
[cn2:00814] [[15019,0],1] posting persistent recv on tag 21 for peer [[WILDCARD],WILDCARD]
[cn2:00814] [[15019,0],1] posting recv
[cn2:00814] [[15019,0],1] posting persistent recv on tag 45 for peer [[WILDCARD],WILDCARD]
[cn2:00814] [[15019,0],1] posting recv
[cn2:00814] [[15019,0],1] posting persistent recv on tag 46 for peer [[WILDCARD],WILDCARD]
[cn2:00814] [[15019,0],1] posting recv
[cn2:00814] [[15019,0],1] posting persistent recv on tag 1 for peer [[WILDCARD],WILDCARD]
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 10
[cn1:14018] [[15019,0],0] connection_handler: working connection (15, 11) 192.168.1.2:44866
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1]:tcp:processing set_peer cmd for interface eth0
[cn2:00814] [[15019,0],1] SET_PEER ADDING PEER [[15019,0],0]
[cn2:00814] [[15019,0],1] set_peer: peer [[15019,0],0] is listening on interface eth0 net 192.168.1.1 port 57056
[cn1:14018] [[15019,0],0] accept_connection: 192.168.1.2:44866
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:10
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called
[cn1:14018] [[15019,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 15
[cn1:14018] [[15019,0],0] waiting for connect ack from UNKNOWN
[cn1:14018] [[15019,0],0] connect ack received from UNKNOWN
[cn1:14018] [[15019,0],0] connect-ack recvd from UNKNOWN
[cn1:14018] [[15019,0],0] mca_oob_tcp_recv_connect: connection from new peer
[cn1:14018] [[15019,0],0] connect-ack header from [[15019,0],1] is okay
[cn1:14018] [[15019,0],0] waiting for connect ack from [[15019,0],1]
[cn1:14018] [[15019,0],0] connect ack received from [[15019,0],1]
[cn1:14018] [[15019,0],0] connect-ack version from [[15019,0],1] matches ours
[cn1:14018] [[15019,0],0] connect-ack [[15019,0],1] authenticated
[cn1:14018] [[15019,0],0] tcp:peer_accept called for peer [[15019,0],1] in state UNKNOWN on socket 15
[cn1:14018] [[15019,0],0] SEND CONNECT ACK
[cn1:14018] [[15019,0],0] send blocking of 47 bytes to socket 15
[cn1:14018] [[15019,0],0] blocking send complete to socket 15
[cn1:14018] [[15019,0],0]-[[15019,0],1] tcp_peer_connected on socket 15
[cn1:14018] [[15019,0],0] routed_binomial_update: [[15019,0],1] --> [[15019,0],1]
[cn1:14018] [[15019,0],0]-[[15019,0],1] accepted: 192.168.1.1 - 192.168.1.2 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:14018] [[15019,0],0] tcp:set_module called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:10
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: initiating connection to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],0]
[cn2:00814] [[15019,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] via interface eth0
[cn2:00814] [[15019,0],1] oob:tcp:peer creating socket to [[15019,0],0]
[cn2:00814] [[15019,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] via interface eth0 on socket 9
[cn2:00814] [[15019,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] on 192.168.1.1:57056 - 0 retries
[cn2:00814] [[15019,0],1] waiting for connect completion to [[15019,0],0] - activating send event
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler CONNECTING
[cn2:00814] [[15019,0],1]:tcp:complete_connect called for peer [[15019,0],0] on socket 9
[cn2:00814] [[15019,0],1] tcp_peer_complete_connect: sending ack to [[15019,0],0]
[cn2:00814] [[15019,0],1] SEND CONNECT ACK
[cn2:00814] fetching data from INTERNAL dstore
[cn2:00814] dstore:hash:fetch: searching for key opal.cred on proc 5279252480
[cn2:00814] dstore_hash:fetch key opal.cred for proc 5279252480 not found
[cn2:00814] [[15019,0],1] send blocking of 47 bytes to socket 9
[cn2:00814] [[15019,0],1] blocking send complete to socket 9
[cn2:00814] [[15019,0],1] tcp_peer_complete_connect: setting read event on connection to [[15019,0],0]
[cn2:00814] [[15019,0],1]:tcp:recv:handler called for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] RECV CONNECT ACK FROM [[15019,0],0] ON SOCKET 9
[cn2:00814] [[15019,0],1] waiting for connect ack from [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 13097
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 13097 BYTES FOR DEST [[15019,0],0] TAG 10
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 13097 bytes from [[15019,0],1] for tag 10
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch from daemon [[15019,0],1]
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch from daemon [[15019,0],1] on node cn2
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch attempting to assign daemon [[15019,0],1] to node cn2
[cn3:21265] mca: base: components_register: registering dstore components
[cn3:21265] mca: base: components_register: found loaded component hash
[cn3:21265] mca: base: components_register: component hash register function successful
[cn3:21265] mca: base: components_open: opening dstore components
[cn3:21265] mca: base: components_open: found loaded component hash
[cn3:21265] mca:dstore:select: checking available component hash
[cn3:21265] mca:dstore:select: Querying component [hash]
[cn3:21265] mca: base: components_register: registering grpcomm components
[cn3:21265] mca: base: components_register: found loaded component bad
[cn3:21265] mca: base: components_register: component bad has no register or open function
[cn3:21265] mca: base: components_register: found loaded component pmi
[cn3:21265] mca: base: components_register: component pmi register function successful
[cn3:21265] mca: base: components_open: opening grpcomm components
[cn1:14018] [[15019,0],0] RECEIVED TOPOLOGY FROM NODE cn2
[cn3:21265] mca: base: components_open: found loaded component bad
[cn3:21265] mca: base: components_open: component bad open function successful
[cn3:21265] mca: base: components_open: found loaded component pmi
[cn3:21265] mca: base: components_open: component pmi open function successful
[cn3:21265] mca:base:select: Auto-selecting grpcomm components
[cn3:21265] mca:base:select:(grpcomm) Querying component [bad]
[cn3:21265] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn3:21265] mca:base:select:(grpcomm) Querying component [pmi]
[cn3:21265] mca:base:select:(grpcomm) Skipping component [pmi]. Query failed to return a module
[cn3:21265] mca:base:select:(grpcomm) Selected component [bad]
[cn3:21265] mca: base: close: component pmi closed
[cn3:21265] mca: base: close: unloading component pmi
[cn3:21265] [[15019,0],2] grpcomm:base:receive start comm
[cn3:21265] [[15019,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn3:21265] [[15019,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 15
[cn3:21265] [[15019,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 32
[cn3:21265] [[15019,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 11
[cn1:14018] Type: Machine Number of child objects: 2
	Name=NULL
	total=25100516KB
	DMIProductName=S5520UR
	DMIProductVersion=....................
	DMIBoardVendor="Intel Corporation"
	DMIBoardName=S5520UR
	DMIBoardVersion=E22554-751
	DMIBoardAssetTag=....................
	DMIChassisVendor=...............................
	DMIChassisType=23
	DMIChassisVersion=..................
	DMIChassisAssetTag=....................
	DMIBIOSVendor="Intel Corp."
	DMIBIOSVersion=S5500.86B.01.00.0054.092820101104
	DMIBIOSDate=09/28/2010
	DMISysVendor="Intel Corporation"
	Backend=Linux
	LinuxCgroup=/
	OSName=Linux
	OSRelease=2.6.32-431.3.1.el6.x86_64
	OSVersion="#1 SMP Fri Jan 3 21:39:27 UTC 2014"
	Architecture=x86_64
	Cpuset:  0x000000ff
	Online:  0x000000ff
	Allowed: 0x000000ff
	Bind CPU proc:   TRUE
	Bind CPU thread: TRUE
	Bind MEM proc:   FALSE
	Bind MEM thread: FALSE
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12517604KB
		total=12517604KB
		Cpuset:  0x00000055
		Online:  0x00000055
		Allowed: 0x00000055
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x00000055
			Online:  0x00000055
			Allowed: 0x00000055
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x00000055
				Online:  0x00000055
				Allowed: 0x00000055
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000001
					Online:  0x00000001
					Allowed: 0x00000001
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000001
						Online:  0x00000001
						Allowed: 0x00000001
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000001
							Online:  0x00000001
							Allowed: 0x00000001
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000001
								Online:  0x00000001
								Allowed: 0x00000001
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000004
					Online:  0x00000004
					Allowed: 0x00000004
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000004
						Online:  0x00000004
						Allowed: 0x00000004
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000004
							Online:  0x00000004
							Allowed: 0x00000004
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000004
								Online:  0x00000004
								Allowed: 0x00000004
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000010
					Online:  0x00000010
					Allowed: 0x00000010
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000010
						Online:  0x00000010
						Allowed: 0x00000010
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000010
							Online:  0x00000010
							Allowed: 0x00000010
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000010
								Online:  0x00000010
								Allowed: 0x00000010
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000040
					Online:  0x00000040
					Allowed: 0x00000040
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000040
						Online:  0x00000040
						Allowed: 0x00000040
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000040
							Online:  0x00000040
							Allowed: 0x00000040
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000040
								Online:  0x00000040
								Allowed: 0x00000040
	Type: NUMANode Number of child objects: 1
		Name=NULL
		local=12582912KB
		total=12582912KB
		Cpuset:  0x000000aa
		Online:  0x000000aa
		Allowed: 0x000000aa
		Type: Socket Number of child objects: 1
			Name=NULL
			CPUModel="Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
			Cpuset:  0x000000aa
			Online:  0x000000aa
			Allowed: 0x000000aa
			Type: L3Cache Number of child objects: 4
				Name=NULL
				size=12288KB
				linesize=64
				ways=16
				Cpuset:  0x000000aa
				Online:  0x000000aa
				Allowed: 0x000000aa
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000002
					Online:  0x00000002
					Allowed: 0x00000002
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000002
						Online:  0x00000002
						Allowed: 0x00000002
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000002
							Online:  0x00000002
							Allowed: 0x00000002
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000002
								Online:  0x00000002
								Allowed: 0x00000002
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000008
					Online:  0x00000008
					Allowed: 0x00000008
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000008
						Online:  0x00000008
						Allowed: 0x00000008
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000008
							Online:  0x00000008
							Allowed: 0x00000008
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000008
								Online:  0x00000008
								Allowed: 0x00000008
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000020
					Online:  0x00000020
					Allowed: 0x00000020
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000020
						Online:  0x00000020
						Allowed: 0x00000020
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000020
							Online:  0x00000020
							Allowed: 0x00000020
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000020
								Online:  0x00000020
								Allowed: 0x00000020
				Type: L2Cache Number of child objects: 1
					Name=NULL
					size=256KB
					linesize=64
					ways=8
					Cpuset:  0x00000080
					Online:  0x00000080
					Allowed: 0x00000080
					Type: L1dCache Number of child objects: 1
						Name=NULL
						size=32KB
						linesize=64
						ways=8
						Cpuset:  0x00000080
						Online:  0x00000080
						Allowed: 0x00000080
						Type: Core Number of child objects: 1
							Name=NULL
							Cpuset:  0x00000080
							Online:  0x00000080
							Allowed: 0x00000080
							Type: PU Number of child objects: 0
								Name=NULL
								Cpuset:  0x00000080
								Online:  0x00000080
								Allowed: 0x00000080
[cn1:14018] [[15019,0],0] TOPOLOGY MATCHES - DISCARDING
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch completed for daemon [[15019,0],1] at contact 984285184.1;tcp://192.168.1.2:49722
[cn1:14018] [[15019,0],0]: set_addr to uri 984285184.1;tcp://192.168.1.2:49722
[cn1:14018] [[15019,0],0]:set_addr checking if peer [[15019,0],1] is reachable via component tcp
[cn1:14018] [[15019,0],0] oob:tcp: working peer [[15019,0],1] address tcp://192.168.1.2:49722
[cn1:14018] [[15019,0],0] PEER [[15019,0],1] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:14018] [[15019,0],0] PASSING ADDR 192.168.1.2 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:14018] [[15019,0],0]:tcp set addr for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]: peer [[15019,0],1] is reachable via component tcp
[cn1:14018] [[15019,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:14018] [[15019,0],0] set_peer: peer [[15019,0],1] is listening on interface eth0 net 192.168.1.2 port 49722
[cn3:21265] [[15019,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn3:21265] [[15019,0],2] routed:binomial rank 0 parent 0 me 2 num_procs 5
[cn3:21265] [[15019,0],2] routed:binomial find children of rank 0
[cn3:21265] [[15019,0],2] routed:binomial find children checking peer 1
[cn3:21265] [[15019,0],2] routed:binomial find children computing tree
[cn3:21265] [[15019,0],2] routed:binomial rank 1 parent 0 me 2 num_procs 5
[cn3:21265] [[15019,0],2] routed:binomial find children of rank 1
[cn3:21265] [[15019,0],2] routed:binomial find children checking peer 3
[cn3:21265] [[15019,0],2] routed:binomial find children computing tree
[cn3:21265] [[15019,0],2] routed:binomial rank 3 parent 1 me 2 num_procs 5
[cn3:21265] [[15019,0],2] routed:binomial find children of rank 3
[cn3:21265] [[15019,0],2] routed:binomial find children checking peer 7
[cn3:21265] [[15019,0],2] routed:binomial find children checking peer 5
[cn3:21265] [[15019,0],2] routed:binomial find children checking peer 2
[cn3:21265] [[15019,0],2] routed:binomial find children computing tree
[cn3:21265] [[15019,0],2] routed:binomial rank 2 parent 0 me 2 num_procs 5
[cn3:21265] [[15019,0],2] routed:binomial find children returning found value 0
[cn3:21265] [[15019,0],2]: parent 0 num_children 0
[cn3:21265] [[15019,0],2] setting up session dir with
	tmpdir: UNDEF
	host cn3
[cn3:21265] procdir: /tmp/openmpi-sessions-artpol@cn3_0/15019/0/2
[cn3:21265] jobdir: /tmp/openmpi-sessions-artpol@cn3_0/15019/0
[cn3:21265] top: openmpi-sessions-artpol@cn3_0
[cn3:21265] tmp: /tmp
[cn3:21265] sess_dir_cleanup: job session dir does not exist
[cn4:17677] [[15019,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 3
[cn4:17677] [[15019,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 21
[cn4:17677] [[15019,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 45
[cn4:17677] [[15019,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 46
[cn4:17677] [[15019,0],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 1
Daemon [[15019,0],3] checking in as pid 17677 on host cn4
[cn4:17677] progress: progress_set_yield_when_idle to false
[cn4:17677] progress: set_event_flag setting to 1
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 10
[cn4:17677] [[15019,0],3] orted: up and running - waiting for commands!
[cn4:17677] [[15019,0],3] posting recv
[cn4:17677] [[15019,0],3] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn4:17677] [[15019,0],3] posting recv
[cn4:17677] [[15019,0],3] posting persistent recv on tag 15 for peer [[WILDCARD],WILDCARD]
[cn4:17677] [[15019,0],3] posting recv
[cn4:17677] [[15019,0],3] posting persistent recv on tag 32 for peer [[WILDCARD],WILDCARD]
[cn4:17677] [[15019,0],3] posting recv
[cn4:17677] [[15019,0],3] posting persistent recv on tag 11 for peer [[WILDCARD],WILDCARD]
[cn4:17677] [[15019,0],3] posting recv
[cn4:17677] [[15019,0],3] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn4:17677] [[15019,0],3]: set_addr to uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn4:17677] [[15019,0],3]:set_addr checking if peer [[15019,0],0] is reachable via component tcp
[cn4:17677] [[15019,0],3] oob:tcp: working peer [[15019,0],0] address tcp://192.168.1.1,192.168.10.2:57056
[cn4:17677] [[15019,0],3] PEER [[15019,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17677] [[15019,0],3] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn4:17677] [[15019,0],3]:tcp set addr for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn4:17677] [[15019,0],3]: peer [[15019,0],0] is reachable via component tcp
[cn4:17677] [[15019,0],3] posting recv
[cn4:17677] [[15019,0],3] posting persistent recv on tag 3 for peer [[WILDCARD],WILDCARD]
[cn4:17677] [[15019,0],3] posting recv
[cn4:17677] [[15019,0],3] posting persistent recv on tag 21 for peer [[WILDCARD],WILDCARD]
[cn4:17677] [[15019,0],3] posting recv
[cn4:17677] [[15019,0],3] posting persistent recv on tag 45 for peer [[WILDCARD],WILDCARD]
[cn4:17677] [[15019,0],3] posting recv
[cn4:17677] [[15019,0],3] posting persistent recv on tag 46 for peer [[WILDCARD],WILDCARD]
[cn4:17677] [[15019,0],3] posting recv
[cn4:17677] [[15019,0],3] posting persistent recv on tag 1 for peer [[WILDCARD],WILDCARD]
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 10
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3]:tcp:processing set_peer cmd for interface eth0
[cn4:17677] [[15019,0],3] SET_PEER ADDING PEER [[15019,0],0]
[cn4:17677] [[15019,0],3] set_peer: peer [[15019,0],0] is listening on interface eth0 net 192.168.1.1 port 57056
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:10
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:10
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: initiating connection to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],0]
[cn4:17677] [[15019,0],3] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] via interface eth0
[cn4:17677] [[15019,0],3] oob:tcp:peer creating socket to [[15019,0],0]
[cn1:14018] [[15019,0],0] mca_oob_tcp_listen_thread: new connection: (16, 11) 192.168.1.4:32927
[cn1:14018] [[15019,0],0] connection_handler: working connection (16, 0) 192.168.1.4:32927
[cn4:17677] [[15019,0],3] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] via interface eth0 on socket 9
[cn4:17677] [[15019,0],3] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] on 192.168.1.1:57056 - 0 retries
[cn1:14018] [[15019,0],0] accept_connection: 192.168.1.4:32927
[cn4:17677] [[15019,0],3] waiting for connect completion to [[15019,0],0] - activating send event
[cn1:14018] [[15019,0],0]:tcp:recv:handler called
[cn1:14018] [[15019,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 16
[cn1:14018] [[15019,0],0] waiting for connect ack from UNKNOWN
[cn1:14018] [[15019,0],0] connect ack received from UNKNOWN
[cn1:14018] [[15019,0],0] connect-ack recvd from UNKNOWN
[cn1:14018] [[15019,0],0] mca_oob_tcp_recv_connect: connection from new peer
[cn1:14018] [[15019,0],0] connect-ack header from [[15019,0],3] is okay
[cn1:14018] [[15019,0],0] waiting for connect ack from [[15019,0],3]
[cn1:14018] [[15019,0],0] connect ack received from [[15019,0],3]
[cn1:14018] [[15019,0],0] connect-ack version from [[15019,0],3] matches ours
[cn1:14018] [[15019,0],0] connect-ack [[15019,0],3] authenticated
[cn1:14018] [[15019,0],0] tcp:peer_accept called for peer [[15019,0],3] in state UNKNOWN on socket 16
[cn1:14018] [[15019,0],0] SEND CONNECT ACK
[cn1:14018] [[15019,0],0] send blocking of 47 bytes to socket 16
[cn1:14018] [[15019,0],0] blocking send complete to socket 16
[cn1:14018] [[15019,0],0]-[[15019,0],3] tcp_peer_connected on socket 16
[cn1:14018] [[15019,0],0] routed_binomial_update: [[15019,0],3] --> [[15019,0],3]
[cn1:14018] [[15019,0],0]-[[15019,0],3] accepted: 192.168.1.1 - 192.168.1.4 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:14018] [[15019,0],0] tcp:set_module called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] tcp:send_handler CONNECTING
[cn4:17677] [[15019,0],3]:tcp:complete_connect called for peer [[15019,0],0] on socket 9
[cn4:17677] [[15019,0],3] tcp_peer_complete_connect: sending ack to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] SEND CONNECT ACK
[cn4:17677] fetching data from INTERNAL dstore
[cn4:17677] dstore:hash:fetch: searching for key opal.cred on proc 13869187072
[cn4:17677] dstore_hash:fetch key opal.cred for proc 13869187072 not found
[cn4:17677] [[15019,0],3] send blocking of 47 bytes to socket 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 96
[cn4:17677] [[15019,0],3] blocking send complete to socket 9
[cn4:17677] [[15019,0],3] tcp_peer_complete_connect: setting read event on connection to [[15019,0],0]
[cn4:17677] [[15019,0],3]:tcp:recv:handler called for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] RECV CONNECT ACK FROM [[15019,0],0] ON SOCKET 9
[cn4:17677] [[15019,0],3] waiting for connect ack from [[15019,0],0]
[cn4:17677] [[15019,0],3] connect ack received from [[15019,0],0]
[cn4:17677] [[15019,0],3] connect-ack recvd from [[15019,0],0]
[cn4:17677] [[15019,0],3] connect-ack header from [[15019,0],0] is okay
[cn4:17677] [[15019,0],3] waiting for connect ack from [[15019,0],0]
[cn4:17677] [[15019,0],3] connect ack received from [[15019,0],0]
[cn4:17677] [[15019,0],3] connect-ack version from [[15019,0],0] matches ours
[cn4:17677] [[15019,0],3] connect-ack [[15019,0],0] authenticated
[cn4:17677] [[15019,0],3]-[[15019,0],0] tcp_peer_connected on socket 9
[cn4:17677] [[15019,0],3] routed_binomial_update: [[15019,0],0] --> [[15019,0],0]
[cn4:17677] [[15019,0],3]-[[15019,0],0] connected: 192.168.1.4 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn4:17677] [[15019,0],3]:tcp:recv:handler starting send/recv events
[cn4:17677] [[15019,0],3] tcp:set_module called for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 96 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] procdir: /tmp/openmpi-sessions-artpol@cn3_0/15019/0/2
[cn3:21265] jobdir: /tmp/openmpi-sessions-artpol@cn3_0/15019/0
[cn3:21265] top: openmpi-sessions-artpol@cn3_0
[cn3:21265] tmp: /tmp
[cn3:21265] [[15019,0],2] routed_binomial: init routes for daemon job [15019,0]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn3:21265] [[15019,0],2] routed_binomial: completed init routes
[cn3:21265] [[15019,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 3
[cn3:21265] [[15019,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 21
[cn3:21265] [[15019,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 45
[cn3:21265] [[15019,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 46
[cn3:21265] [[15019,0],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 1
Daemon [[15019,0],2] checking in as pid 21265 on host cn3
[cn3:21265] progress: progress_set_yield_when_idle to false
[cn3:21265] progress: set_event_flag setting to 1
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 10
[cn3:21265] [[15019,0],2] orted: up and running - waiting for commands!
[cn3:21265] [[15019,0],2] posting recv
[cn3:21265] [[15019,0],2] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn3:21265] [[15019,0],2] posting recv
[cn3:21265] [[15019,0],2] posting persistent recv on tag 15 for peer [[WILDCARD],WILDCARD]
[cn3:21265] [[15019,0],2] posting recv
[cn3:21265] [[15019,0],2] posting persistent recv on tag 32 for peer [[WILDCARD],WILDCARD]
[cn3:21265] [[15019,0],2] posting recv
[cn3:21265] [[15019,0],2] posting persistent recv on tag 11 for peer [[WILDCARD],WILDCARD]
[cn3:21265] [[15019,0],2] posting recv
[cn3:21265] [[15019,0],2] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn3:21265] [[15019,0],2]: set_addr to uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn3:21265] [[15019,0],2]:set_addr checking if peer [[15019,0],0] is reachable via component tcp
[cn3:21265] [[15019,0],2] oob:tcp: working peer [[15019,0],0] address tcp://192.168.1.1,192.168.10.2:57056
[cn3:21265] [[15019,0],2] PEER [[15019,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn3:21265] [[15019,0],2] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21265] [[15019,0],2]:tcp set addr for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn3:21265] [[15019,0],2]: peer [[15019,0],0] is reachable via component tcp
[cn3:21265] [[15019,0],2] posting recv
[cn3:21265] [[15019,0],2] posting persistent recv on tag 3 for peer [[WILDCARD],WILDCARD]
[cn3:21265] [[15019,0],2] posting recv
[cn3:21265] [[15019,0],2] posting persistent recv on tag 21 for peer [[WILDCARD],WILDCARD]
[cn3:21265] [[15019,0],2] posting recv
[cn3:21265] [[15019,0],2] posting persistent recv on tag 45 for peer [[WILDCARD],WILDCARD]
[cn3:21265] [[15019,0],2] posting recv
[cn3:21265] [[15019,0],2] posting persistent recv on tag 46 for peer [[WILDCARD],WILDCARD]
[cn3:21265] [[15019,0],2] posting recv
[cn3:21265] [[15019,0],2] posting persistent recv on tag 1 for peer [[WILDCARD],WILDCARD]
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 10
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2]:tcp:processing set_peer cmd for interface eth0
[cn3:21265] [[15019,0],2] SET_PEER ADDING PEER [[15019,0],0]
[cn3:21265] [[15019,0],2] set_peer: peer [[15019,0],0] is listening on interface eth0 net 192.168.1.1 port 57056
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:10
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:10
[cn1:14018] [[15019,0],0] mca_oob_tcp_listen_thread: new connection: (17, 11) 192.168.1.3:58458
[cn1:14018] [[15019,0],0] connection_handler: working connection (17, 11) 192.168.1.3:58458
[cn1:14018] [[15019,0],0] accept_connection: 192.168.1.3:58458
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called
[cn3:21265] [[15019,0],2] tcp:send_nb: initiating connection to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 17
[cn1:14018] [[15019,0],0] waiting for connect ack from UNKNOWN
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],0]
[cn3:21265] [[15019,0],2] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] via interface eth0
[cn3:21265] [[15019,0],2] oob:tcp:peer creating socket to [[15019,0],0]
[cn1:14018] [[15019,0],0] connect ack received from UNKNOWN
[cn1:14018] [[15019,0],0] connect-ack recvd from UNKNOWN
[cn3:21265] [[15019,0],2] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] via interface eth0 on socket 9
[cn1:14018] [[15019,0],0] mca_oob_tcp_recv_connect: connection from new peer
[cn3:21265] [[15019,0],2] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] on 192.168.1.1:57056 - 0 retries
[cn1:14018] [[15019,0],0] connect-ack header from [[15019,0],2] is okay
[cn1:14018] [[15019,0],0] waiting for connect ack from [[15019,0],2]
[cn3:21265] [[15019,0],2] waiting for connect completion to [[15019,0],0] - activating send event
[cn1:14018] [[15019,0],0] connect ack received from [[15019,0],2]
[cn1:14018] [[15019,0],0] connect-ack version from [[15019,0],2] matches ours
[cn1:14018] [[15019,0],0] connect-ack [[15019,0],2] authenticated
[cn1:14018] [[15019,0],0] tcp:peer_accept called for peer [[15019,0],2] in state UNKNOWN on socket 17
[cn1:14018] [[15019,0],0] SEND CONNECT ACK
[cn1:14018] [[15019,0],0] send blocking of 47 bytes to socket 17
[cn1:14018] [[15019,0],0] blocking send complete to socket 17
[cn1:14018] [[15019,0],0]-[[15019,0],2] tcp_peer_connected on socket 17
[cn1:14018] [[15019,0],0] routed_binomial_update: [[15019,0],2] --> [[15019,0],2]
[cn1:14018] [[15019,0],0]-[[15019,0],2] accepted: 192.168.1.1 - 192.168.1.3 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:14018] [[15019,0],0] tcp:set_module called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler CONNECTING
[cn3:21265] [[15019,0],2]:tcp:complete_connect called for peer [[15019,0],0] on socket 9
[cn3:21265] [[15019,0],2] tcp_peer_complete_connect: sending ack to [[15019,0],0]
[cn3:21265] [[15019,0],2] SEND CONNECT ACK
[cn3:21265] fetching data from INTERNAL dstore
[cn3:21265] dstore:hash:fetch: searching for key opal.cred on proc 9574219776
[cn3:21265] dstore_hash:fetch key opal.cred for proc 9574219776 not found
[cn3:21265] [[15019,0],2] send blocking of 47 bytes to socket 9
[cn3:21265] [[15019,0],2] blocking send complete to socket 9
[cn3:21265] [[15019,0],2] tcp_peer_complete_connect: setting read event on connection to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2]:tcp:recv:handler called for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] RECV CONNECT ACK FROM [[15019,0],0] ON SOCKET 9
[cn3:21265] [[15019,0],2] waiting for connect ack from [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] connect ack received from [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] connect-ack recvd from [[15019,0],0]
[cn3:21265] [[15019,0],2] connect-ack header from [[15019,0],0] is okay
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 96
[cn3:21265] [[15019,0],2] waiting for connect ack from [[15019,0],0]
[cn3:21265] [[15019,0],2] connect ack received from [[15019,0],0]
[cn3:21265] [[15019,0],2] connect-ack version from [[15019,0],0] matches ours
[cn3:21265] [[15019,0],2] connect-ack [[15019,0],0] authenticated
[cn3:21265] [[15019,0],2]-[[15019,0],0] tcp_peer_connected on socket 9
[cn3:21265] [[15019,0],2] routed_binomial_update: [[15019,0],0] --> [[15019,0],0]
[cn3:21265] [[15019,0],2]-[[15019,0],0] connected: 192.168.1.3 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn3:21265] [[15019,0],2]:tcp:recv:handler starting send/recv events
[cn3:21265] [[15019,0],2] tcp:set_module called for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 96 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 96 BYTES FOR DEST [[15019,0],0] TAG 10
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 96 bytes from [[15019,0],4] for tag 10
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch from daemon [[15019,0],4]
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch from daemon [[15019,0],4] on node cn5
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch attempting to assign daemon [[15019,0],4] to node cn5
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch completed for daemon [[15019,0],4] at contact 984285184.4;tcp://192.168.1.5:41719
[cn1:14018] [[15019,0],0]: set_addr to uri 984285184.4;tcp://192.168.1.5:41719
[cn1:14018] [[15019,0],0]:set_addr checking if peer [[15019,0],4] is reachable via component tcp
[cn1:14018] [[15019,0],0] oob:tcp: working peer [[15019,0],4] address tcp://192.168.1.5:41719
[cn1:14018] [[15019,0],0] PEER [[15019,0],4] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:14018] [[15019,0],0] PASSING ADDR 192.168.1.5 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:14018] [[15019,0],0]:tcp set addr for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]: peer [[15019,0],4] is reachable via component tcp
[cn1:14018] [[15019,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:14018] [[15019,0],0] set_peer: peer [[15019,0],4] is listening on interface eth0 net 192.168.1.5 port 41719
[cn2:00814] [[15019,0],1] connect ack received from [[15019,0],0]
[cn2:00814] [[15019,0],1] connect-ack recvd from [[15019,0],0]
[cn2:00814] [[15019,0],1] connect-ack header from [[15019,0],0] is okay
[cn2:00814] [[15019,0],1] waiting for connect ack from [[15019,0],0]
[cn2:00814] [[15019,0],1] connect ack received from [[15019,0],0]
[cn2:00814] [[15019,0],1] connect-ack version from [[15019,0],0] matches ours
[cn2:00814] [[15019,0],1] connect-ack [[15019,0],0] authenticated
[cn2:00814] [[15019,0],1]-[[15019,0],0] tcp_peer_connected on socket 9
[cn2:00814] [[15019,0],1] routed_binomial_update: [[15019,0],0] --> [[15019,0],0]
[cn2:00814] [[15019,0],1]-[[15019,0],0] connected: 192.168.1.2 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn2:00814] [[15019,0],1]:tcp:recv:handler starting send/recv events
[cn2:00814] [[15019,0],1] tcp:set_module called for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 13097 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 96 BYTES FOR DEST [[15019,0],0] TAG 10
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 96 bytes from [[15019,0],3] for tag 10
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch from daemon [[15019,0],3]
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch from daemon [[15019,0],3] on node cn4
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch attempting to assign daemon [[15019,0],3] to node cn4
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch completed for daemon [[15019,0],3] at contact 984285184.3;tcp://192.168.1.4:43236
[cn1:14018] [[15019,0],0]: set_addr to uri 984285184.3;tcp://192.168.1.4:43236
[cn1:14018] [[15019,0],0]:set_addr checking if peer [[15019,0],3] is reachable via component tcp
[cn1:14018] [[15019,0],0] oob:tcp: working peer [[15019,0],3] address tcp://192.168.1.4:43236
[cn1:14018] [[15019,0],0] PEER [[15019,0],3] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:14018] [[15019,0],0] PASSING ADDR 192.168.1.4 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:14018] [[15019,0],0]:tcp set addr for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]: peer [[15019,0],3] is reachable via component tcp
[cn1:14018] [[15019,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:14018] [[15019,0],0] set_peer: peer [[15019,0],3] is listening on interface eth0 net 192.168.1.4 port 43236
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 96 BYTES FOR DEST [[15019,0],0] TAG 10
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 96 bytes from [[15019,0],2] for tag 10
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch from daemon [[15019,0],2]
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch from daemon [[15019,0],2] on node cn3
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch attempting to assign daemon [[15019,0],2] to node cn3
[cn1:14018] [[15019,0],0] plm:base:orted_report_launch completed for daemon [[15019,0],2] at contact 984285184.2;tcp://192.168.1.3:38382
[cn1:14018] [[15019,0],0]: set_addr to uri 984285184.2;tcp://192.168.1.3:38382
[cn1:14018] [[15019,0],0]:set_addr checking if peer [[15019,0],2] is reachable via component tcp
[cn1:14018] [[15019,0],0] oob:tcp: working peer [[15019,0],2] address tcp://192.168.1.3:38382
[cn1:14018] [[15019,0],0] PEER [[15019,0],2] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:14018] [[15019,0],0] PASSING ADDR 192.168.1.3 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:14018] [[15019,0],0]:tcp set addr for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]: peer [[15019,0],2] is reachable via component tcp
[cn1:14018] [[15019,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:14018] [[15019,0],0] set_peer: peer [[15019,0],2] is listening on interface eth0 net 192.168.1.3 port 38382
[cn1:14018] mca:rmaps: mapping job [15019,1]
[cn1:14018] mca:rmaps: creating new map for job [15019,1]
[cn1:14018] mca:rmaps: nprocs 0
[cn1:14018] mca:rmaps mapping not given - using bycore
[cn1:14018] mca:rmaps:ppr: job [15019,1] not using ppr mapper
[cn1:14018] [[15019,0],0] rmaps:seq mapping job [15019,1]
[cn1:14018] mca:rmaps:seq: job [15019,1] not using seq mapper
[cn1:14018] mca:rmaps:resilient: cannot perform initial map of job [15019,1] - no fault groups
[cn1:14018] mca:rmaps:mindist: job [15019,1] not using mindist mapper
[cn1:14018] mca:rmaps:rr: mapping job [15019,1]
[cn1:14018] [[15019,0],0] Starting with 5 nodes in list
[cn1:14018] [[15019,0],0] Filtering thru apps
[cn1:14018] [[15019,0],0] Retained 5 nodes in list
[cn1:14018] [[15019,0],0] node cn1 has 2 slots available
[cn1:14018] [[15019,0],0] node cn2 has 2 slots available
[cn1:14018] [[15019,0],0] node cn3 has 2 slots available
[cn1:14018] [[15019,0],0] node cn4 has 2 slots available
[cn1:14018] [[15019,0],0] node cn5 has 2 slots available
[cn1:14018] AVAILABLE NODES FOR MAPPING:
[cn1:14018]     node: cn1 daemon: 0
[cn1:14018]     node: cn2 daemon: 1
[cn1:14018]     node: cn3 daemon: 2
[cn1:14018]     node: cn4 daemon: 3
[cn1:14018]     node: cn5 daemon: 4
[cn1:14018] [[15019,0],0] Starting bookmark at node cn1
[cn1:14018] [[15019,0],0] Starting at node cn1
[cn1:14018] mca:rmaps:rr: mapping no-span by Core for job [15019,1] slots 10 num_procs 10
[cn1:14018] mca:rmaps:rr: found 8 Core objects on node cn1
[cn1:14018] mca:rmaps:rr: calculated nprocs 2
[cn1:14018] mca:rmaps:rr: assigning nprocs 2
[cn1:14018] mca:rmaps:rr: assigning proc to object 0
[cn1:14018] mca:rmaps:rr: assigning proc to object 1
[cn1:14018] mca:rmaps:rr: found 8 Core objects on node cn2
[cn1:14018] mca:rmaps:rr: calculated nprocs 2
[cn1:14018] mca:rmaps:rr: assigning nprocs 2
[cn1:14018] mca:rmaps:rr: assigning proc to object 0
[cn1:14018] mca:rmaps:rr: assigning proc to object 1
[cn1:14018] mca:rmaps:rr: found 8 Core objects on node cn3
[cn1:14018] mca:rmaps:rr: calculated nprocs 2
[cn1:14018] mca:rmaps:rr: assigning nprocs 2
[cn1:14018] mca:rmaps:rr: assigning proc to object 0
[cn1:14018] mca:rmaps:rr: assigning proc to object 1
[cn1:14018] mca:rmaps:rr: found 8 Core objects on node cn4
[cn1:14018] mca:rmaps:rr: calculated nprocs 2
[cn1:14018] mca:rmaps:rr: assigning nprocs 2
[cn1:14018] mca:rmaps:rr: assigning proc to object 0
[cn1:14018] mca:rmaps:rr: assigning proc to object 1
[cn1:14018] mca:rmaps:rr: found 8 Core objects on node cn5
[cn1:14018] mca:rmaps:rr: calculated nprocs 2
[cn1:14018] mca:rmaps:rr: assigning nprocs 2
[cn1:14018] mca:rmaps:rr: assigning proc to object 0
[cn1:14018] mca:rmaps:rr: assigning proc to object 1
[cn1:14018] mca:rmaps:base: computing vpids by slot for job [15019,1]
[cn1:14018] mca:rmaps:base: assigning rank 0 to node cn1
[cn1:14018] mca:rmaps:base: assigning rank 1 to node cn1
[cn1:14018] mca:rmaps:base: assigning rank 2 to node cn2
[cn1:14018] mca:rmaps:base: assigning rank 3 to node cn2
[cn1:14018] mca:rmaps:base: assigning rank 4 to node cn3
[cn1:14018] mca:rmaps:base: assigning rank 5 to node cn3
[cn1:14018] mca:rmaps:base: assigning rank 6 to node cn4
[cn1:14018] mca:rmaps:base: assigning rank 7 to node cn4
[cn1:14018] mca:rmaps:base: assigning rank 8 to node cn5
[cn1:14018] mca:rmaps:base: assigning rank 9 to node cn5
[cn1:14018] [[15019,0],0] rmaps:base:compute_usage
[cn1:14018] mca:rmaps: compute bindings for job [15019,1] with policy CORE
[cn1:14018] mca:rmaps: bindings for job [15019,1] - bind in place
[cn1:14018] mca:rmaps: bind in place for job [15019,1] with bindings CORE
[cn1:14018] [[15019,0],0] reset_usage: node cn1 has 2 procs on it
[cn1:14018] [[15019,0],0] reset_usage: ignoring proc [[15019,1],0]
[cn1:14018] [[15019,0],0] reset_usage: ignoring proc [[15019,1],1]
[cn1:14018] BINDING PROC [[15019,1],0] TO Core NUMBER 0
[cn1:14018] [[15019,0],0] BOUND PROC [[15019,1],0] TO 0[Core:0] on node cn1
[cn1:14018] BINDING PROC [[15019,1],1] TO Core NUMBER 1
[cn1:14018] [[15019,0],0] BOUND PROC [[15019,1],1] TO 2[Core:1] on node cn1
[cn1:14018] [[15019,0],0] reset_usage: node cn2 has 2 procs on it
[cn1:14018] [[15019,0],0] reset_usage: ignoring proc [[15019,1],2]
[cn1:14018] [[15019,0],0] reset_usage: ignoring proc [[15019,1],3]
[cn1:14018] BINDING PROC [[15019,1],2] TO Core NUMBER 0
[cn1:14018] [[15019,0],0] BOUND PROC [[15019,1],2] TO 0[Core:0] on node cn2
[cn1:14018] BINDING PROC [[15019,1],3] TO Core NUMBER 1
[cn1:14018] [[15019,0],0] BOUND PROC [[15019,1],3] TO 2[Core:1] on node cn2
[cn1:14018] [[15019,0],0] reset_usage: node cn3 has 2 procs on it
[cn1:14018] [[15019,0],0] reset_usage: ignoring proc [[15019,1],4]
[cn1:14018] [[15019,0],0] reset_usage: ignoring proc [[15019,1],5]
[cn1:14018] BINDING PROC [[15019,1],4] TO Core NUMBER 0
[cn1:14018] [[15019,0],0] BOUND PROC [[15019,1],4] TO 0[Core:0] on node cn3
[cn1:14018] BINDING PROC [[15019,1],5] TO Core NUMBER 1
[cn1:14018] [[15019,0],0] BOUND PROC [[15019,1],5] TO 2[Core:1] on node cn3
[cn1:14018] [[15019,0],0] reset_usage: node cn4 has 2 procs on it
[cn1:14018] [[15019,0],0] reset_usage: ignoring proc [[15019,1],6]
[cn1:14018] [[15019,0],0] reset_usage: ignoring proc [[15019,1],7]
[cn1:14018] BINDING PROC [[15019,1],6] TO Core NUMBER 0
[cn1:14018] [[15019,0],0] BOUND PROC [[15019,1],6] TO 0[Core:0] on node cn4
[cn1:14018] BINDING PROC [[15019,1],7] TO Core NUMBER 1
[cn1:14018] [[15019,0],0] BOUND PROC [[15019,1],7] TO 2[Core:1] on node cn4
[cn1:14018] [[15019,0],0] reset_usage: node cn5 has 2 procs on it
[cn1:14018] [[15019,0],0] reset_usage: ignoring proc [[15019,1],8]
[cn1:14018] [[15019,0],0] reset_usage: ignoring proc [[15019,1],9]
[cn1:14018] BINDING PROC [[15019,1],8] TO Core NUMBER 0
[cn1:14018] [[15019,0],0] BOUND PROC [[15019,1],8] TO 0[Core:0] on node cn5
[cn1:14018] BINDING PROC [[15019,1],9] TO Core NUMBER 1
[cn1:14018] [[15019,0],0] BOUND PROC [[15019,1],9] TO 2[Core:1] on node cn5
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 0 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial 0 found child 1
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 1 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 0
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 1
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 1 parent 0 me 1 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial 1 found child 3
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 3 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 0
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 1
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 1 parent 0 me 3 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 1
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 3
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 3 parent 1 me 3 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 1
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 1
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 0
[cn1:14018] [[15019,0],0] routed:binomial 0 found child 2
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 2 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 0
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 1
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 1 parent 0 me 2 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 1
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 3
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 3 parent 1 me 2 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 3
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 7
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 5
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 2
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 2 parent 0 me 2 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 0
[cn1:14018] [[15019,0],0] routed:binomial 0 found child 4
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 0
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 1
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 1 parent 0 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 1
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 3
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 3 parent 1 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 3
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 7
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 5
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 2
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 2 parent 0 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 2
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 6
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 4
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 4 parent 0 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 0
[cn1:14018] [[15019,0],0]: parent 0 num_children 3
[cn1:14018] [[15019,0],0]: 	child 1
[cn1:14018] [[15019,0],0]: 		relation 3
[cn1:14018] [[15019,0],0]: 	child 2
[cn1:14018] [[15019,0],0]: 	child 4
[cn1:14018] [[15019,0],0] plm:base:launch_apps for job [15019,1]
[cn1:14018] [[15019,0],0] grpcomm:bad:xcast sent to job [15019,0] tag 1
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],0] at tag 15
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],0] at tag 15
[cn1:14018] [[15019,0],0] rml_send_iovec_to_self at tag 15
[cn1:14018] [[15019,0],0] rml_send_to_self callback executing for tag 15
[cn1:14018] [[15019,0],0] message received 2634 bytes from [[15019,0],0] for tag 15
[cn1:14018] [[15019,0],0] grpcomm:xcast:recv: with 2634 bytes
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 0 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial 0 found child 1
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 1 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 0
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 1
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 1 parent 0 me 1 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial 1 found child 3
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 3 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 0
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 1
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 1 parent 0 me 3 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 1
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 3
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 3 parent 1 me 3 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 1
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 1
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 0
[cn1:14018] [[15019,0],0] routed:binomial 0 found child 2
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 2 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 0
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 1
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 1 parent 0 me 2 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 1
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 3
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 3 parent 1 me 2 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 3
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 7
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 5
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 2
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 2 parent 0 me 2 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 0
[cn1:14018] [[15019,0],0] routed:binomial 0 found child 4
[cn1:14018] [[15019,0],0] routed:binomial rank 0 parent 0 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 0
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 1
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 1 parent 0 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 1
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 3
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 3 parent 1 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 3
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 7
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 5
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 2
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 2 parent 0 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children of rank 2
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 6
[cn1:14018] [[15019,0],0] routed:binomial find children checking peer 4
[cn1:14018] [[15019,0],0] routed:binomial find children computing tree
[cn1:14018] [[15019,0],0] routed:binomial rank 4 parent 0 me 4 num_procs 5
[cn1:14018] [[15019,0],0] routed:binomial find children returning found value 0
[cn1:14018] [[15019,0],0]: parent 0 num_children 3
[cn1:14018] [[15019,0],0]: 	child 1
[cn1:14018] [[15019,0],0]: 		relation 3
[cn1:14018] [[15019,0],0]: 	child 2
[cn1:14018] [[15019,0],0]: 	child 4
[cn1:14018] [[15019,0],0] routed_binomial: init routes for HNP job [15019,0]
[cn1:14018] [[15019,0],0] rml:base:update:contact:info got uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn1:14018] [[15019,0],0] rml:base:update:contact:info got uri 984285184.1;tcp://192.168.1.2:49722
[cn1:14018] [[15019,0],0] rml:base:update:contact:info got uri 984285184.2;tcp://192.168.1.3:38382
[cn1:14018] [[15019,0],0] rml:base:update:contact:info got uri 984285184.3;tcp://192.168.1.4:43236
[cn1:14018] [[15019,0],0] rml:base:update:contact:info got uri 984285184.4;tcp://192.168.1.5:41719
[cn1:14018] [[15019,0],0] orte:daemon:send_relay sending relay msg to [[15019,0],1]
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],1] at tag 15
[cn1:14018] [[15019,0],0] orte:daemon:send_relay sending relay msg to [[15019,0],2]
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],2] at tag 15
[cn1:14018] [[15019,0],0] orte:daemon:send_relay sending relay msg to [[15019,0],4]
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],4] at tag 15
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],0] at tag 1
[cn1:14018] [[15019,0],0]: set_addr to uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn1:14018] [[15019,0],0]:set_addr peer [[15019,0],0] is me
[cn1:14018] [[15019,0],0]: set_addr to uri 984285184.1;tcp://192.168.1.2:49722
[cn1:14018] [[15019,0],0]:set_addr checking if peer [[15019,0],1] is reachable via component tcp
[cn1:14018] [[15019,0],0] oob:tcp: working peer [[15019,0],1] address tcp://192.168.1.2:49722
[cn1:14018] [[15019,0],0] PEER [[15019,0],1] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:14018] [[15019,0],0] PASSING ADDR 192.168.1.2 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:14018] [[15019,0],0]:tcp set addr for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]: peer [[15019,0],1] is reachable via component tcp
[cn1:14018] [[15019,0],0]: set_addr to uri 984285184.2;tcp://192.168.1.3:38382
[cn1:14018] [[15019,0],0]:set_addr checking if peer [[15019,0],2] is reachable via component tcp
[cn1:14018] [[15019,0],0] oob:tcp: working peer [[15019,0],2] address tcp://192.168.1.3:38382
[cn1:14018] [[15019,0],0] PEER [[15019,0],2] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:14018] [[15019,0],0] PASSING ADDR 192.168.1.3 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:14018] [[15019,0],0]:tcp set addr for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]: peer [[15019,0],2] is reachable via component tcp
[cn1:14018] [[15019,0],0]: set_addr to uri 984285184.3;tcp://192.168.1.4:43236
[cn1:14018] [[15019,0],0]:set_addr checking if peer [[15019,0],3] is reachable via component tcp
[cn1:14018] [[15019,0],0] oob:tcp: working peer [[15019,0],3] address tcp://192.168.1.4:43236
[cn1:14018] [[15019,0],0] PEER [[15019,0],3] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:14018] [[15019,0],0] PASSING ADDR 192.168.1.4 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:14018] [[15019,0],0]:tcp set addr for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]: peer [[15019,0],3] is reachable via component tcp
[cn1:14018] [[15019,0],0]: set_addr to uri 984285184.4;tcp://192.168.1.5:41719
[cn1:14018] [[15019,0],0]:set_addr checking if peer [[15019,0],4] is reachable via component tcp
[cn1:14018] [[15019,0],0] oob:tcp: working peer [[15019,0],4] address tcp://192.168.1.5:41719
[cn1:14018] [[15019,0],0] PEER [[15019,0],4] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:14018] [[15019,0],0] PASSING ADDR 192.168.1.5 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:14018] [[15019,0],0]:tcp set addr for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]: peer [[15019,0],4] is reachable via component tcp
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],1] at tag 15
[cn1:14018] [[15019,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],2] at tag 15
[cn1:14018] [[15019,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],4] at tag 15
[cn1:14018] [[15019,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],0] at tag 1
[cn1:14018] [[15019,0],0] rml_send_iovec_to_self at tag 1
[cn1:14018] [[15019,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:14018] [[15019,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:14018] [[15019,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:14018] [[15019,0],0]:tcp:processing set_peer cmd for interface eth0
[cn1:14018] [[15019,0],0] oob:base:send to target [[15019,0],1]
[cn1:14018] [[15019,0],0] oob:base:send known transport for peer [[15019,0],1]
[cn1:14018] [[15019,0],0] oob:tcp:send_nb to peer [[15019,0],1]:15
[cn1:14018] [[15019,0],0] tcp:send_nb to peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],1]
[cn1:14018] [[15019,0],0] oob:base:send to target [[15019,0],2]
[cn1:14018] [[15019,0],0] oob:base:send known transport for peer [[15019,0],2]
[cn1:14018] [[15019,0],0] oob:tcp:send_nb to peer [[15019,0],2]:15
[cn1:14018] [[15019,0],0] tcp:send_nb to peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],2]
[cn1:14018] [[15019,0],0] oob:base:send to target [[15019,0],4]
[cn1:14018] [[15019,0],0] oob:base:send known transport for peer [[15019,0],4]
[cn1:14018] [[15019,0],0] oob:tcp:send_nb to peer [[15019,0],4]:15
[cn1:14018] [[15019,0],0] tcp:send_nb to peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],4]
[cn1:14018] [[15019,0],0] rml_send_to_self callback executing for tag 1
[cn1:14018] [[15019,0],0] message received 2634 bytes from [[15019,0],0] for tag 1
[cn1:14018] [[15019,0],0] orted:comm:process_commands() Processing Command: ORTE_DAEMON_ADD_LOCAL_PROCS
[cn1:14018] [[15019,0],0] orted_cmd: received add_local_procs
[cn1:14018] [[15019,0],0] PROGRESSING COLL id 0
[cn1:14018] [[15019,0],0] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn1:14018] [[15019,0],0] PROGRESSING COLL id 1
[cn1:14018] [[15019,0],0] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn1:14018] [[15019,0],0] PROGRESSING COLL id 2
[cn1:14018] [[15019,0],0] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],1]:15
[cn1:14018] [[15019,0],0] routed_binomial_get([[15019,0],1]) --> [[15019,0],1]
[cn1:14018] [[15019,0],0] tcp:send_nb: already connected to [[15019,0],1] - queueing for send
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],1]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],2]:15
[cn1:14018] [[15019,0],0] routed_binomial_get([[15019,0],2]) --> [[15019,0],2]
[cn1:14018] [[15019,0],0] tcp:send_nb: already connected to [[15019,0],2] - queueing for send
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],2]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],4]:15
[cn1:14018] [[15019,0],0] routed_binomial_get([[15019,0],4]) --> [[15019,0],4]
[cn1:14018] [[15019,0],0] tcp:send_nb: already connected to [[15019,0],4] - queueing for send
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],4]
[cn1:14018] [[15019,0],0] tcp:send_handler called to send to peer [[15019,0],1]
[cn1:14018] [[15019,0],0] tcp:send_handler SENDING TO [[15019,0],1]
[cn1:14018] [[15019,0],0] MESSAGE SEND COMPLETE TO [[15019,0],1] OF 2634 BYTES ON SOCKET 15
[cn1:14018] [[15019,0],0]-[[15019,0],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] tcp:send_handler called to send to peer [[15019,0],2]
[cn1:14018] [[15019,0],0] tcp:send_handler SENDING TO [[15019,0],2]
[cn1:14018] [[15019,0],0] MESSAGE SEND COMPLETE TO [[15019,0],2] OF 2634 BYTES ON SOCKET 17
[cn1:14018] [[15019,0],0]-[[15019,0],2] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] tcp:send_handler called to send to peer [[15019,0],4]
[cn1:14018] [[15019,0],0] tcp:send_handler SENDING TO [[15019,0],4]
[cn1:14018] [[15019,0],0] MESSAGE SEND COMPLETE TO [[15019,0],4] OF 2634 BYTES ON SOCKET 14
[cn1:14018] [[15019,0],0]-[[15019,0],4] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2]:tcp:recv:handler called for peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1]:tcp:recv:handler called for peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1]:tcp:recv:handler allocate data region of size 2634
[cn5:14014] [[15019,0],4]:tcp:recv:handler called for peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:tcp:recv:handler allocate data region of size 2634
[cn3:21265] [[15019,0],2]:tcp:recv:handler allocate data region of size 2634
[cn2:00814] [[15019,0],1]:tcp:recv:handler called for peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] RECVD COMPLETE MESSAGE FROM [[15019,0],0] (ORIGIN [[15019,0],0]) OF 2634 BYTES FOR DEST [[15019,0],1] TAG 15
[cn2:00814] [[15019,0],1] DELIVERING TO RML
[cn2:00814] [[15019,0],1] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] message received 2634 bytes from [[15019,0],0] for tag 15
[cn2:00814] [[15019,0],1] grpcomm:xcast:recv: with 2634 bytes
[cn2:00814] [[15019,0],1] grpcomm:base:xcast updating daemon nidmap
[cn5:14014] [[15019,0],4]:tcp:recv:handler called for peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] RECVD COMPLETE MESSAGE FROM [[15019,0],0] (ORIGIN [[15019,0],0]) OF 2634 BYTES FOR DEST [[15019,0],4] TAG 15
[cn5:14014] [[15019,0],4] DELIVERING TO RML
[cn5:14014] [[15019,0],4] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] message received 2634 bytes from [[15019,0],0] for tag 15
[cn3:21265] [[15019,0],2]:tcp:recv:handler called for peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] RECVD COMPLETE MESSAGE FROM [[15019,0],0] (ORIGIN [[15019,0],0]) OF 2634 BYTES FOR DEST [[15019,0],2] TAG 15
[cn3:21265] [[15019,0],2] DELIVERING TO RML
[cn3:21265] [[15019,0],2] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] message received 2634 bytes from [[15019,0],0] for tag 15
[cn5:14014] [[15019,0],4] grpcomm:xcast:recv: with 2634 bytes
[cn2:00814] [[15019,0],1] routed:binomial rank 0 parent 0 me 1 num_procs 5
[cn2:00814] [[15019,0],1] routed:binomial find children of rank 0
[cn2:00814] [[15019,0],1] routed:binomial find children checking peer 1
[cn2:00814] [[15019,0],1] routed:binomial find children computing tree
[cn2:00814] [[15019,0],1] routed:binomial rank 1 parent 0 me 1 num_procs 5
[cn2:00814] [[15019,0],1] routed:binomial 1 found child 3
[cn2:00814] [[15019,0],1] routed:binomial rank 0 parent 0 me 3 num_procs 5
[cn2:00814] [[15019,0],1] routed:binomial find children of rank 0
[cn2:00814] [[15019,0],1] routed:binomial find children checking peer 1
[cn2:00814] [[15019,0],1] routed:binomial find children computing tree
[cn2:00814] [[15019,0],1] routed:binomial rank 1 parent 0 me 3 num_procs 5
[cn2:00814] [[15019,0],1] routed:binomial find children of rank 1
[cn2:00814] [[15019,0],1] routed:binomial find children checking peer 3
[cn2:00814] [[15019,0],1] routed:binomial find children computing tree
[cn5:14014] [[15019,0],4] grpcomm:base:xcast updating daemon nidmap
[cn3:21265] [[15019,0],2] grpcomm:xcast:recv: with 2634 bytes
[cn3:21265] [[15019,0],2] grpcomm:base:xcast updating daemon nidmap
[cn5:14014] [[15019,0],4] routed:binomial rank 0 parent 0 me 4 num_procs 5
[cn5:14014] [[15019,0],4] routed:binomial find children of rank 0
[cn5:14014] [[15019,0],4] routed:binomial find children checking peer 1
[cn5:14014] [[15019,0],4] routed:binomial find children computing tree
[cn5:14014] [[15019,0],4] routed:binomial rank 1 parent 0 me 4 num_procs 5
[cn5:14014] [[15019,0],4] routed:binomial find children of rank 1
[cn5:14014] [[15019,0],4] routed:binomial find children checking peer 3
[cn5:14014] [[15019,0],4] routed:binomial find children computing tree
[cn5:14014] [[15019,0],4] routed:binomial rank 3 parent 1 me 4 num_procs 5
[cn5:14014] [[15019,0],4] routed:binomial find children of rank 3
[cn5:14014] [[15019,0],4] routed:binomial find children checking peer 7
[cn5:14014] [[15019,0],4] routed:binomial find children checking peer 5
[cn5:14014] [[15019,0],4] routed:binomial find children checking peer 2
[cn5:14014] [[15019,0],4] routed:binomial find children computing tree
[cn2:00814] [[15019,0],1] routed:binomial rank 3 parent 1 me 3 num_procs 5
[cn2:00814] [[15019,0],1] routed:binomial find children returning found value 1
[cn2:00814] [[15019,0],1] routed:binomial find children returning found value 1
[cn2:00814] [[15019,0],1] routed:binomial find children returning found value 0
[cn2:00814] [[15019,0],1]: parent 0 num_children 1
[cn2:00814] [[15019,0],1]: 	child 3
[cn2:00814] [[15019,0],1] routed_binomial: init routes for daemon job [15019,0]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn2:00814] [[15019,0],1] rml:base:update:contact:info got uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn5:14014] [[15019,0],4] routed:binomial rank 2 parent 0 me 4 num_procs 5
[cn5:14014] [[15019,0],4] routed:binomial find children of rank 2
[cn5:14014] [[15019,0],4] routed:binomial find children checking peer 6
[cn5:14014] [[15019,0],4] routed:binomial find children checking peer 4
[cn5:14014] [[15019,0],4] routed:binomial find children computing tree
[cn5:14014] [[15019,0],4] routed:binomial rank 4 parent 0 me 4 num_procs 5
[cn5:14014] [[15019,0],4] routed:binomial find children returning found value 0
[cn5:14014] [[15019,0],4]: parent 0 num_children 0
[cn5:14014] [[15019,0],4] routed_binomial: init routes for daemon job [15019,0]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn3:21265] [[15019,0],2] routed:binomial rank 0 parent 0 me 2 num_procs 5
[cn3:21265] [[15019,0],2] routed:binomial find children of rank 0
[cn3:21265] [[15019,0],2] routed:binomial find children checking peer 1
[cn3:21265] [[15019,0],2] routed:binomial find children computing tree
[cn3:21265] [[15019,0],2] routed:binomial rank 1 parent 0 me 2 num_procs 5
[cn3:21265] [[15019,0],2] routed:binomial find children of rank 1
[cn3:21265] [[15019,0],2] routed:binomial find children checking peer 3
[cn3:21265] [[15019,0],2] routed:binomial find children computing tree
[cn3:21265] [[15019,0],2] routed:binomial rank 3 parent 1 me 2 num_procs 5
[cn3:21265] [[15019,0],2] routed:binomial find children of rank 3
[cn3:21265] [[15019,0],2] routed:binomial find children checking peer 7
[cn3:21265] [[15019,0],2] routed:binomial find children checking peer 5
[cn3:21265] [[15019,0],2] routed:binomial find children checking peer 2
[cn3:21265] [[15019,0],2] routed:binomial find children computing tree
[cn2:00814] [[15019,0],1] rml:base:update:contact:info got uri 984285184.1;tcp://192.168.1.2:49722
[cn2:00814] [[15019,0],1] rml:base:update:contact:info got uri 984285184.2;tcp://192.168.1.3:38382
[cn2:00814] [[15019,0],1] rml:base:update:contact:info got uri 984285184.3;tcp://192.168.1.4:43236
[cn2:00814] [[15019,0],1] rml:base:update:contact:info got uri 984285184.4;tcp://192.168.1.5:41719
[cn2:00814] [[15019,0],1] orte:daemon:send_relay sending relay msg to [[15019,0],3]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],3] at tag 15
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],1] at tag 1
[cn5:14014] [[15019,0],4] rml:base:update:contact:info got uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn5:14014] [[15019,0],4] rml:base:update:contact:info got uri 984285184.1;tcp://192.168.1.2:49722
[cn5:14014] [[15019,0],4] rml:base:update:contact:info got uri 984285184.2;tcp://192.168.1.3:38382
[cn5:14014] [[15019,0],4] rml:base:update:contact:info got uri 984285184.3;tcp://192.168.1.4:43236
[cn5:14014] [[15019,0],4] rml:base:update:contact:info got uri 984285184.4;tcp://192.168.1.5:41719
[cn5:14014] [[15019,0],4] orte:daemon:send_relay - recipient list is empty!
[cn2:00814] [[15019,0],1]: set_addr to uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn2:00814] [[15019,0],1]:set_addr checking if peer [[15019,0],0] is reachable via component tcp
[cn2:00814] [[15019,0],1] oob:tcp: working peer [[15019,0],0] address tcp://192.168.1.1,192.168.10.2:57056
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],4] at tag 1
[cn5:14014] [[15019,0],4]: set_addr to uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn5:14014] [[15019,0],4]:set_addr checking if peer [[15019,0],0] is reachable via component tcp
[cn5:14014] [[15019,0],4] oob:tcp: working peer [[15019,0],0] address tcp://192.168.1.1,192.168.10.2:57056
[cn2:00814] [[15019,0],1] PEER [[15019,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:14014] [[15019,0],4] PEER [[15019,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:14014] [[15019,0],4] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00814] [[15019,0],1] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00814] [[15019,0],1]:tcp set addr for peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:tcp set addr for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn5:14014] [[15019,0],4] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn2:00814] [[15019,0],1]: peer [[15019,0],0] is reachable via component tcp
[cn2:00814] [[15019,0],1]: set_addr to uri 984285184.1;tcp://192.168.1.2:49722
[cn5:14014] [[15019,0],4]: peer [[15019,0],0] is reachable via component tcp
[cn5:14014] [[15019,0],4]: set_addr to uri 984285184.1;tcp://192.168.1.2:49722
[cn3:21265] [[15019,0],2] routed:binomial rank 2 parent 0 me 2 num_procs 5
[cn3:21265] [[15019,0],2] routed:binomial find children returning found value 0
[cn3:21265] [[15019,0],2]: parent 0 num_children 0
[cn3:21265] [[15019,0],2] routed_binomial: init routes for daemon job [15019,0]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn3:21265] [[15019,0],2] rml:base:update:contact:info got uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn3:21265] [[15019,0],2] rml:base:update:contact:info got uri 984285184.1;tcp://192.168.1.2:49722
[cn2:00814] [[15019,0],1]:set_addr peer [[15019,0],1] is me
[cn2:00814] [[15019,0],1]: set_addr to uri 984285184.2;tcp://192.168.1.3:38382
[cn5:14014] [[15019,0],4]:set_addr checking if peer [[15019,0],1] is reachable via component tcp
[cn5:14014] [[15019,0],4] oob:tcp: working peer [[15019,0],1] address tcp://192.168.1.2:49722
[cn3:21265] [[15019,0],2] rml:base:update:contact:info got uri 984285184.2;tcp://192.168.1.3:38382
[cn3:21265] [[15019,0],2] rml:base:update:contact:info got uri 984285184.3;tcp://192.168.1.4:43236
[cn3:21265] [[15019,0],2] rml:base:update:contact:info got uri 984285184.4;tcp://192.168.1.5:41719
[cn3:21265] [[15019,0],2] orte:daemon:send_relay - recipient list is empty!
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],2] at tag 1
[cn3:21265] [[15019,0],2]: set_addr to uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn3:21265] [[15019,0],2]:set_addr checking if peer [[15019,0],0] is reachable via component tcp
[cn2:00814] [[15019,0],1]:set_addr checking if peer [[15019,0],2] is reachable via component tcp
[cn2:00814] [[15019,0],1] oob:tcp: working peer [[15019,0],2] address tcp://192.168.1.3:38382
[cn5:14014] [[15019,0],4] PEER [[15019,0],1] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:14014] [[15019,0],4] PASSING ADDR 192.168.1.2 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21265] [[15019,0],2] oob:tcp: working peer [[15019,0],0] address tcp://192.168.1.1,192.168.10.2:57056
[cn2:00814] [[15019,0],1] PEER [[15019,0],2] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:14014] [[15019,0],4]:tcp set addr for peer [[15019,0],1]
[cn5:14014] [[15019,0],4]: peer [[15019,0],1] is reachable via component tcp
[cn3:21265] [[15019,0],2] PEER [[15019,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn3:21265] [[15019,0],2] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21265] [[15019,0],2]:tcp set addr for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] PASSING ADDR 192.168.1.3 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00814] [[15019,0],1]:tcp set addr for peer [[15019,0],2]
[cn5:14014] [[15019,0],4]: set_addr to uri 984285184.2;tcp://192.168.1.3:38382
[cn5:14014] [[15019,0],4]:set_addr checking if peer [[15019,0],2] is reachable via component tcp
[cn3:21265] [[15019,0],2] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn5:14014] [[15019,0],4] oob:tcp: working peer [[15019,0],2] address tcp://192.168.1.3:38382
[cn3:21265] [[15019,0],2]: peer [[15019,0],0] is reachable via component tcp
[cn3:21265] [[15019,0],2]: set_addr to uri 984285184.1;tcp://192.168.1.2:49722
[cn5:14014] [[15019,0],4] PEER [[15019,0],2] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:14014] [[15019,0],4] PASSING ADDR 192.168.1.3 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn5:14014] [[15019,0],4]:tcp set addr for peer [[15019,0],2]
[cn5:14014] [[15019,0],4]: peer [[15019,0],2] is reachable via component tcp
[cn5:14014] [[15019,0],4]: set_addr to uri 984285184.3;tcp://192.168.1.4:43236
[cn5:14014] [[15019,0],4]:set_addr checking if peer [[15019,0],3] is reachable via component tcp
[cn5:14014] [[15019,0],4] oob:tcp: working peer [[15019,0],3] address tcp://192.168.1.4:43236
[cn5:14014] [[15019,0],4] PEER [[15019,0],3] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn2:00814] [[15019,0],1]: peer [[15019,0],2] is reachable via component tcp
[cn2:00814] [[15019,0],1]: set_addr to uri 984285184.3;tcp://192.168.1.4:43236
[cn2:00814] [[15019,0],1]:set_addr checking if peer [[15019,0],3] is reachable via component tcp
[cn5:14014] [[15019,0],4] PASSING ADDR 192.168.1.4 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn5:14014] [[15019,0],4]:tcp set addr for peer [[15019,0],3]
[cn3:21265] [[15019,0],2]:set_addr checking if peer [[15019,0],1] is reachable via component tcp
[cn2:00814] [[15019,0],1] oob:tcp: working peer [[15019,0],3] address tcp://192.168.1.4:43236
[cn5:14014] [[15019,0],4]: peer [[15019,0],3] is reachable via component tcp
[cn5:14014] [[15019,0],4]: set_addr to uri 984285184.4;tcp://192.168.1.5:41719
[cn5:14014] [[15019,0],4]:set_addr peer [[15019,0],4] is me
[cn3:21265] [[15019,0],2] oob:tcp: working peer [[15019,0],1] address tcp://192.168.1.2:49722
[cn2:00814] [[15019,0],1] PEER [[15019,0],3] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],4] at tag 1
[cn5:14014] [[15019,0],4] rml_send_iovec_to_self at tag 1
[cn3:21265] [[15019,0],2] PEER [[15019,0],1] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn3:21265] [[15019,0],2] PASSING ADDR 192.168.1.2 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21265] [[15019,0],2]:tcp set addr for peer [[15019,0],1]
[cn2:00814] [[15019,0],1] PASSING ADDR 192.168.1.4 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00814] [[15019,0],1]:tcp set addr for peer [[15019,0],3]
[cn5:14014] [[15019,0],4]:tcp:processing set_peer cmd for interface eth0
[cn3:21265] [[15019,0],2]: peer [[15019,0],1] is reachable via component tcp
[cn3:21265] [[15019,0],2]: set_addr to uri 984285184.2;tcp://192.168.1.3:38382
[cn3:21265] [[15019,0],2]:set_addr peer [[15019,0],2] is me
[cn3:21265] [[15019,0],2]: set_addr to uri 984285184.3;tcp://192.168.1.4:43236
[cn2:00814] [[15019,0],1]: peer [[15019,0],3] is reachable via component tcp
[cn2:00814] [[15019,0],1]: set_addr to uri 984285184.4;tcp://192.168.1.5:41719
[cn2:00814] [[15019,0],1]:set_addr checking if peer [[15019,0],4] is reachable via component tcp
[cn5:14014] [[15019,0],4]:tcp:processing set_peer cmd for interface eth0
[cn5:14014] [[15019,0],4] SET_PEER ADDING PEER [[15019,0],1]
[cn3:21265] [[15019,0],2]:set_addr checking if peer [[15019,0],3] is reachable via component tcp
[cn3:21265] [[15019,0],2] oob:tcp: working peer [[15019,0],3] address tcp://192.168.1.4:43236
[cn2:00814] [[15019,0],1] oob:tcp: working peer [[15019,0],4] address tcp://192.168.1.5:41719
[cn5:14014] [[15019,0],4] set_peer: peer [[15019,0],1] is listening on interface eth0 net 192.168.1.2 port 49722
[cn5:14014] [[15019,0],4]:tcp:processing set_peer cmd for interface eth0
[cn3:21265] [[15019,0],2] PEER [[15019,0],3] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn2:00814] [[15019,0],1] PEER [[15019,0],4] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:14014] [[15019,0],4] SET_PEER ADDING PEER [[15019,0],2]
[cn5:14014] [[15019,0],4] set_peer: peer [[15019,0],2] is listening on interface eth0 net 192.168.1.3 port 38382
[cn5:14014] [[15019,0],4]:tcp:processing set_peer cmd for interface eth0
[cn3:21265] [[15019,0],2] PASSING ADDR 192.168.1.4 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21265] [[15019,0],2]:tcp set addr for peer [[15019,0],3]
[cn3:21265] [[15019,0],2]: peer [[15019,0],3] is reachable via component tcp
[cn2:00814] [[15019,0],1] PASSING ADDR 192.168.1.5 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00814] [[15019,0],1]:tcp set addr for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] SET_PEER ADDING PEER [[15019,0],3]
[cn5:14014] [[15019,0],4] set_peer: peer [[15019,0],3] is listening on interface eth0 net 192.168.1.4 port 43236
[cn3:21265] [[15019,0],2]: set_addr to uri 984285184.4;tcp://192.168.1.5:41719
[cn3:21265] [[15019,0],2]:set_addr checking if peer [[15019,0],4] is reachable via component tcp
[cn2:00814] [[15019,0],1]: peer [[15019,0],4] is reachable via component tcp
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],3] at tag 15
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] rml_send_to_self callback executing for tag 1
[cn5:14014] [[15019,0],4] message received 2634 bytes from [[15019,0],4] for tag 1
[cn3:21265] [[15019,0],2] oob:tcp: working peer [[15019,0],4] address tcp://192.168.1.5:41719
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],1] at tag 1
[cn2:00814] [[15019,0],1] rml_send_iovec_to_self at tag 1
[cn5:14014] [[15019,0],4] orted:comm:process_commands() Processing Command: ORTE_DAEMON_ADD_LOCAL_PROCS
[cn5:14014] [[15019,0],4] orted_cmd: received add_local_procs
[cn3:21265] [[15019,0],2] PEER [[15019,0],4] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn3:21265] [[15019,0],2] PASSING ADDR 192.168.1.5 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21265] [[15019,0],2]:tcp set addr for peer [[15019,0],4]
[cn2:00814] [[15019,0],1]:tcp:processing set_peer cmd for interface eth0
[cn2:00814] [[15019,0],1]:tcp:processing set_peer cmd for interface eth0
[cn3:21265] [[15019,0],2]: peer [[15019,0],4] is reachable via component tcp
[cn2:00814] [[15019,0],1] SET_PEER ADDING PEER [[15019,0],2]
[cn2:00814] [[15019,0],1] set_peer: peer [[15019,0],2] is listening on interface eth0 net 192.168.1.3 port 38382
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],2] at tag 1
[cn3:21265] [[15019,0],2] rml_send_iovec_to_self at tag 1
[cn2:00814] [[15019,0],1]:tcp:processing set_peer cmd for interface eth0
[cn2:00814] [[15019,0],1] SET_PEER ADDING PEER [[15019,0],3]
[cn2:00814] [[15019,0],1] set_peer: peer [[15019,0],3] is listening on interface eth0 net 192.168.1.4 port 43236
[cn2:00814] [[15019,0],1]:tcp:processing set_peer cmd for interface eth0
[cn2:00814] [[15019,0],1] SET_PEER ADDING PEER [[15019,0],4]
[cn2:00814] [[15019,0],1] set_peer: peer [[15019,0],4] is listening on interface eth0 net 192.168.1.5 port 41719
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],3]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],3]:15
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],3]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],3]
[cn2:00814] [[15019,0],1] rml_send_to_self callback executing for tag 1
[cn2:00814] [[15019,0],1] message received 2634 bytes from [[15019,0],1] for tag 1
[cn2:00814] [[15019,0],1] orted:comm:process_commands() Processing Command: ORTE_DAEMON_ADD_LOCAL_PROCS
[cn2:00814] [[15019,0],1] orted_cmd: received add_local_procs
[cn5:14014] [[15019,0],4] PROGRESSING COLL id 0
[cn5:14014] [[15019,0],4] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn5:14014] [[15019,0],4] PROGRESSING COLL id 1
[cn5:14014] [[15019,0],4] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn5:14014] [[15019,0],4] PROGRESSING COLL id 2
[cn5:14014] [[15019,0],4] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn2:00814] [[15019,0],1] PROGRESSING COLL id 0
[cn2:00814] [[15019,0],1] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn2:00814] [[15019,0],1] PROGRESSING COLL id 1
[cn2:00814] [[15019,0],1] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn2:00814] [[15019,0],1] PROGRESSING COLL id 2
[cn2:00814] [[15019,0],1] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],3]:15
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],3]) --> [[15019,0],3]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],3]
[cn2:00814] [[15019,0],1] tcp:send_nb: initiating connection to [[15019,0],3]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],3]
[cn2:00814] [[15019,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],3] via interface eth0
[cn2:00814] [[15019,0],1] oob:tcp:peer creating socket to [[15019,0],3]
[cn3:21265] [[15019,0],2]:tcp:processing set_peer cmd for interface eth0
[cn3:21265] [[15019,0],2]:tcp:processing set_peer cmd for interface eth0
[cn3:21265] [[15019,0],2] SET_PEER ADDING PEER [[15019,0],1]
[cn3:21265] [[15019,0],2] set_peer: peer [[15019,0],1] is listening on interface eth0 net 192.168.1.2 port 49722
[cn2:00814] [[15019,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],3] via interface eth0 on socket 10
[cn2:00814] [[15019,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],3] on 192.168.1.4:43236 - 0 retries
[cn2:00814] [[15019,0],1] waiting for connect completion to [[15019,0],3] - activating send event
[cn3:21265] [[15019,0],2]:tcp:processing set_peer cmd for interface eth0
[cn3:21265] [[15019,0],2] SET_PEER ADDING PEER [[15019,0],3]
[cn3:21265] [[15019,0],2] set_peer: peer [[15019,0],3] is listening on interface eth0 net 192.168.1.4 port 43236
[cn3:21265] [[15019,0],2]:tcp:processing set_peer cmd for interface eth0
[cn3:21265] [[15019,0],2] SET_PEER ADDING PEER [[15019,0],4]
[cn3:21265] [[15019,0],2] set_peer: peer [[15019,0],4] is listening on interface eth0 net 192.168.1.5 port 41719
[cn3:21265] [[15019,0],2] rml_send_to_self callback executing for tag 1
[cn3:21265] [[15019,0],2] message received 2634 bytes from [[15019,0],2] for tag 1
[cn3:21265] [[15019,0],2] orted:comm:process_commands() Processing Command: ORTE_DAEMON_ADD_LOCAL_PROCS
[cn3:21265] [[15019,0],2] orted_cmd: received add_local_procs
[cn3:21265] [[15019,0],2] PROGRESSING COLL id 0
[cn3:21265] [[15019,0],2] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn3:21265] [[15019,0],2] PROGRESSING COLL id 1
[cn3:21265] [[15019,0],2] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn3:21265] [[15019,0],2] PROGRESSING COLL id 2
[cn3:21265] [[15019,0],2] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn4:17677] [[15019,0],3] connection_event_handler: working connection (10, 115) 192.168.1.2:50126
[cn4:17677] [[15019,0],3] accept_connection: 192.168.1.2:50126
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 5
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 5
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],3]
[cn2:00814] [[15019,0],1] tcp:send_handler CONNECTING
[cn2:00814] [[15019,0],1]:tcp:complete_connect called for peer [[15019,0],3] on socket 10
[cn2:00814] [[15019,0],1] tcp_peer_complete_connect: sending ack to [[15019,0],3]
[cn2:00814] [[15019,0],1] SEND CONNECT ACK
[cn2:00814] [[15019,0],1] send blocking of 47 bytes to socket 10
[cn2:00814] [[15019,0],1] blocking send complete to socket 10
[cn2:00814] [[15019,0],1] tcp_peer_complete_connect: setting read event on connection to [[15019,0],3]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 5
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 5
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:5
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:5
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3]:tcp:recv:handler called
[cn4:17677] [[15019,0],3] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 10
[cn4:17677] [[15019,0],3] waiting for connect ack from UNKNOWN
[cn4:17677] [[15019,0],3] connect ack received from UNKNOWN
[cn4:17677] [[15019,0],3] connect-ack recvd from UNKNOWN
[cn4:17677] [[15019,0],3] mca_oob_tcp_recv_connect: connection from new peer
[cn4:17677] [[15019,0],3] connect-ack header from [[15019,0],1] is okay
[cn4:17677] [[15019,0],3] waiting for connect ack from [[15019,0],1]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:5
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:5
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] connect ack received from [[15019,0],1]
[cn4:17677] [[15019,0],3] connect-ack version from [[15019,0],1] matches ours
[cn4:17677] [[15019,0],3] connect-ack [[15019,0],1] authenticated
[cn2:00814] [[15019,0],1]:tcp:recv:handler called for peer [[15019,0],3]
[cn2:00814] [[15019,0],1] RECV CONNECT ACK FROM [[15019,0],3] ON SOCKET 10
[cn2:00814] [[15019,0],1] waiting for connect ack from [[15019,0],3]
[cn2:00814] [[15019,0],1] connect ack received from [[15019,0],3]
[cn2:00814] [[15019,0],1] connect-ack recvd from [[15019,0],3]
[cn2:00814] [[15019,0],1] connect-ack header from [[15019,0],3] is okay
[cn2:00814] [[15019,0],1] waiting for connect ack from [[15019,0],3]
[cn2:00814] [[15019,0],1] connect ack received from [[15019,0],3]
[cn2:00814] [[15019,0],1] connect-ack version from [[15019,0],3] matches ours
[cn2:00814] [[15019,0],1] connect-ack [[15019,0],3] authenticated
[cn2:00814] [[15019,0],1]-[[15019,0],3] tcp_peer_connected on socket 10
[cn2:00814] [[15019,0],1] routed_binomial_update: [[15019,0],3] --> [[15019,0],3]
[cn4:17677] [[15019,0],3] tcp:peer_accept called for peer [[15019,0],1] in state UNKNOWN on socket 10
[cn4:17677] [[15019,0],3] SEND CONNECT ACK
[cn4:17677] [[15019,0],3] send blocking of 47 bytes to socket 10
[cn2:00814] [[15019,0],1]-[[15019,0],3] connected: 192.168.1.2 - 192.168.1.4 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn2:00814] [[15019,0],1]:tcp:recv:handler starting send/recv events
[cn2:00814] [[15019,0],1] tcp:set_module called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] blocking send complete to socket 10
[cn4:17677] [[15019,0],3]-[[15019,0],1] tcp_peer_connected on socket 10
[cn4:17677] [[15019,0],3] routed_binomial_update: [[15019,0],1] --> [[15019,0],1]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],3]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],3]
[cn4:17677] [[15019,0],3]-[[15019,0],1] accepted: 192.168.1.4 - 192.168.1.2 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn4:17677] [[15019,0],3] tcp:set_module called for peer [[15019,0],1]
[cn4:17677] [[15019,0],3]:tcp:recv:handler called for peer [[15019,0],1]
[cn4:17677] [[15019,0],3]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],3] OF 2634 BYTES ON SOCKET 10
[cn2:00814] [[15019,0],1]-[[15019,0],3] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]:tcp:recv:handler allocate data region of size 2634
[cn4:17677] [[15019,0],3]:tcp:recv:handler called for peer [[15019,0],1]
[cn4:17677] [[15019,0],3]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 2634 BYTES FOR DEST [[15019,0],3] TAG 15
[cn4:17677] [[15019,0],3] DELIVERING TO RML
[cn4:17677] [[15019,0],3] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] message received 2634 bytes from [[15019,0],1] for tag 15
[cn4:17677] [[15019,0],3] grpcomm:xcast:recv: with 2634 bytes
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 5
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 5
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:5
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:5
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] grpcomm:base:xcast updating daemon nidmap
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] routed:binomial rank 0 parent 0 me 3 num_procs 5
[cn4:17677] [[15019,0],3] routed:binomial find children of rank 0
[cn4:17677] [[15019,0],3] routed:binomial find children checking peer 1
[cn4:17677] [[15019,0],3] routed:binomial find children computing tree
[cn4:17677] [[15019,0],3] routed:binomial rank 1 parent 0 me 3 num_procs 5
[cn4:17677] [[15019,0],3] routed:binomial find children of rank 1
[cn4:17677] [[15019,0],3] routed:binomial find children checking peer 3
[cn4:17677] [[15019,0],3] routed:binomial find children computing tree
[cn4:17677] [[15019,0],3] routed:binomial rank 3 parent 1 me 3 num_procs 5
[cn4:17677] [[15019,0],3] routed:binomial find children returning found value 1
[cn4:17677] [[15019,0],3] routed:binomial find children returning found value 1
[cn4:17677] [[15019,0],3]: parent 1 num_children 0
[cn4:17677] [[15019,0],3] routed_binomial: init routes for daemon job [15019,0]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml:base:update:contact:info got uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn4:17677] [[15019,0],3] rml:base:update:contact:info got uri 984285184.1;tcp://192.168.1.2:49722
[cn4:17677] [[15019,0],3] rml:base:update:contact:info got uri 984285184.2;tcp://192.168.1.3:38382
[cn4:17677] [[15019,0],3] rml:base:update:contact:info got uri 984285184.3;tcp://192.168.1.4:43236
[cn4:17677] [[15019,0],3] rml:base:update:contact:info got uri 984285184.4;tcp://192.168.1.5:41719
[cn4:17677] [[15019,0],3] orte:daemon:send_relay - recipient list is empty!
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],3] at tag 1
[cn4:17677] [[15019,0],3]: set_addr to uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn4:17677] [[15019,0],3]:set_addr checking if peer [[15019,0],0] is reachable via component tcp
[cn4:17677] [[15019,0],3] oob:tcp: working peer [[15019,0],0] address tcp://192.168.1.1,192.168.10.2:57056
[cn4:17677] [[15019,0],3] PEER [[15019,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17677] [[15019,0],3] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn4:17677] [[15019,0],3]:tcp set addr for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] UNFOUND KERNEL INDEX -13 FOR ADDRESS 192.168.10.2
[cn4:17677] [[15019,0],3]: peer [[15019,0],0] is reachable via component tcp
[cn4:17677] [[15019,0],3]: set_addr to uri 984285184.1;tcp://192.168.1.2:49722
[cn4:17677] [[15019,0],3]:set_addr checking if peer [[15019,0],1] is reachable via component tcp
[cn4:17677] [[15019,0],3] oob:tcp: working peer [[15019,0],1] address tcp://192.168.1.2:49722
[cn4:17677] [[15019,0],3] PEER [[15019,0],1] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17677] [[15019,0],3] PASSING ADDR 192.168.1.2 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn4:17677] [[15019,0],3]:tcp set addr for peer [[15019,0],1]
[cn4:17677] [[15019,0],3]: peer [[15019,0],1] is reachable via component tcp
[cn4:17677] [[15019,0],3]: set_addr to uri 984285184.2;tcp://192.168.1.3:38382
[cn4:17677] [[15019,0],3]:set_addr checking if peer [[15019,0],2] is reachable via component tcp
[cn4:17677] [[15019,0],3] oob:tcp: working peer [[15019,0],2] address tcp://192.168.1.3:38382
[cn4:17677] [[15019,0],3] PEER [[15019,0],2] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17677] [[15019,0],3] PASSING ADDR 192.168.1.3 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn4:17677] [[15019,0],3]:tcp set addr for peer [[15019,0],2]
[cn4:17677] [[15019,0],3]: peer [[15019,0],2] is reachable via component tcp
[cn4:17677] [[15019,0],3]: set_addr to uri 984285184.3;tcp://192.168.1.4:43236
[cn4:17677] [[15019,0],3]:set_addr peer [[15019,0],3] is me
[cn4:17677] [[15019,0],3]: set_addr to uri 984285184.4;tcp://192.168.1.5:41719
[cn4:17677] [[15019,0],3]:set_addr checking if peer [[15019,0],4] is reachable via component tcp
[cn4:17677] [[15019,0],3] oob:tcp: working peer [[15019,0],4] address tcp://192.168.1.5:41719
[cn4:17677] [[15019,0],3] PEER [[15019,0],4] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17677] [[15019,0],3] PASSING ADDR 192.168.1.5 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn4:17677] [[15019,0],3]:tcp set addr for peer [[15019,0],4]
[cn4:17677] [[15019,0],3]: peer [[15019,0],4] is reachable via component tcp
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],3] at tag 1
[cn4:17677] [[15019,0],3] rml_send_iovec_to_self at tag 1
[cn4:17677] [[15019,0],3]:tcp:processing set_peer cmd for interface eth0
[cn4:17677] [[15019,0],3]:tcp:processing set_peer cmd for interface eth0
[cn4:17677] [[15019,0],3] set_peer: peer [[15019,0],1] is listening on interface eth0 net 192.168.1.2 port 49722
[cn4:17677] [[15019,0],3]:tcp:processing set_peer cmd for interface eth0
[cn4:17677] [[15019,0],3] SET_PEER ADDING PEER [[15019,0],2]
[cn4:17677] [[15019,0],3] set_peer: peer [[15019,0],2] is listening on interface eth0 net 192.168.1.3 port 38382
[cn4:17677] [[15019,0],3]:tcp:processing set_peer cmd for interface eth0
[cn4:17677] [[15019,0],3] SET_PEER ADDING PEER [[15019,0],4]
[cn4:17677] [[15019,0],3] set_peer: peer [[15019,0],4] is listening on interface eth0 net 192.168.1.5 port 41719
[cn4:17677] [[15019,0],3] rml_send_to_self callback executing for tag 1
[cn4:17677] [[15019,0],3] message received 2634 bytes from [[15019,0],3] for tag 1
[cn4:17677] [[15019,0],3] orted:comm:process_commands() Processing Command: ORTE_DAEMON_ADD_LOCAL_PROCS
[cn4:17677] [[15019,0],3] orted_cmd: received add_local_procs
[cn4:17677] [[15019,0],3] PROGRESSING COLL id 0
[cn4:17677] [[15019,0],3] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn4:17677] [[15019,0],3] PROGRESSING COLL id 1
[cn4:17677] [[15019,0],3] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn4:17677] [[15019,0],3] PROGRESSING COLL id 2
[cn4:17677] [[15019,0],3] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 311 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 311 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 311 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 311 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 81 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 260 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 5
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 5
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:5
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:5
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 311 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 779 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 779 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 263 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 779 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 500 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 76 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 121 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 500 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 76 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 192 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 500 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 76 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 311 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 311 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 440 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 78 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 500 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 247 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 395 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 172 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 401 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 88 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 659 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 263 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 121 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 304 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 439 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 137 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 401 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 88 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 172 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 172 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 401 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 401 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 276 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 717 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 305 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 370 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 662 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 442 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 670 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 247 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 442 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 76 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 880 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 537 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 263 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 304 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 263 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 304 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1972 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 520 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 164 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 172 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 176 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 401 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 717 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 374 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 537 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1135 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1135 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 276 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 69 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 370 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 96 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1329 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 821 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 104 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 783 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 616 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 551 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 221 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 192 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1374 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1280 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn5:14014] [[15019,0],4] connection_event_handler: working connection (12, 0) 192.168.1.5:38492
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] accept_connection: 192.168.1.5:38492
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 108 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 787 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 106 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4]:tcp:recv:handler called
[cn5:14014] [[15019,0],4] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 12
[cn5:14014] [[15019,0],4] waiting for connect ack from UNKNOWN
[cn5:14014] [[15019,0],4] connect ack received from UNKNOWN
[cn5:14014] [[15019,0],4] connect-ack recvd from UNKNOWN
[cn5:14014] [[15019,0],4] mca_oob_tcp_recv_connect: connection from new peer
[cn5:14014] [[15019,0],4] connect-ack header from [[15019,1],8] is okay
[cn5:14014] [[15019,0],4] waiting for connect ack from [[15019,1],8]
[cn5:14014] [[15019,0],4] connect ack received from [[15019,1],8]
[cn5:14014] [[15019,0],4] connect-ack version from [[15019,1],8] matches ours
[cn5:14014] [[15019,0],4] connect-ack [[15019,1],8] authenticated
[cn5:14014] [[15019,0],4] tcp:peer_accept called for peer [[15019,1],8] in state UNKNOWN on socket 12
[cn5:14014] [[15019,0],4] SEND CONNECT ACK
[cn5:14014] [[15019,0],4] send blocking of 47 bytes to socket 12
[cn5:14014] [[15019,0],4] blocking send complete to socket 12
[cn5:14014] [[15019,0],4]-[[15019,1],8] tcp_peer_connected on socket 12
[cn5:14014] [[15019,0],4] routed_binomial_update: [[15019,1],8] --> [[15019,1],8]
[cn5:14014] [[15019,0],4]-[[15019,1],8] accepted: 192.168.1.5 - 192.168.1.5 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 645 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:set_module called for peer [[15019,1],8]
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4]:tcp:recv:handler called for peer [[15019,1],8]
[cn5:14014] [[15019,0],4]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:tcp:recv:handler allocate data region of size 61
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 195 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 126 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 443 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 520 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1256 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 221 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1462 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 104 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 228 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 845 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 858 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 102 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 105 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 995 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1319 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 374 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] connection_event_handler: working connection (13, 11) 192.168.1.5:38493
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] accept_connection: 192.168.1.5:38493
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 276 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 104 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 743 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4]:tcp:recv:handler called
[cn5:14014] [[15019,0],4] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 13
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 443 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] waiting for connect ack from UNKNOWN
[cn5:14014] [[15019,0],4] connect ack received from UNKNOWN
[cn5:14014] [[15019,0],4] connect-ack recvd from UNKNOWN
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] mca_oob_tcp_recv_connect: connection from new peer
[cn5:14014] [[15019,0],4] connect-ack header from [[15019,1],9] is okay
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1369 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] waiting for connect ack from [[15019,1],9]
[cn5:14014] [[15019,0],4] connect ack received from [[15019,1],9]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] connect-ack version from [[15019,1],9] matches ours
[cn5:14014] [[15019,0],4] connect-ack [[15019,1],9] authenticated
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 102 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] tcp:peer_accept called for peer [[15019,1],9] in state UNKNOWN on socket 13
[cn5:14014] [[15019,0],4] SEND CONNECT ACK
[cn5:14014] [[15019,0],4] send blocking of 47 bytes to socket 13
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] blocking send complete to socket 13
[cn5:14014] [[15019,0],4]-[[15019,1],9] tcp_peer_connected on socket 13
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_update: [[15019,1],9] --> [[15019,1],9]
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]-[[15019,1],9] accepted: 192.168.1.5 - 192.168.1.5 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 374 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:set_module called for peer [[15019,1],9]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 108 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 195 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1149 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 121 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4]:tcp:recv:handler called for peer [[15019,1],9]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:tcp:recv:handler allocate data region of size 61
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 738 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 190 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 343 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 105 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 192 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 738 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 190 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 105 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 520 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 290 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1212 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 159 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 121 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1056 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 335 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] connection_event_handler: working connection (12, 0) 192.168.1.3:49701
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] accept_connection: 192.168.1.3:49701
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 793 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 578 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2]:tcp:recv:handler called
[cn3:21265] [[15019,0],2] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 12
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 664 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] waiting for connect ack from UNKNOWN
[cn3:21265] [[15019,0],2] connect ack received from UNKNOWN
[cn3:21265] [[15019,0],2] connect-ack recvd from UNKNOWN
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] mca_oob_tcp_recv_connect: connection from new peer
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] connect-ack header from [[15019,1],4] is okay
[cn3:21265] [[15019,0],2] waiting for connect ack from [[15019,1],4]
[cn3:21265] [[15019,0],2] connect ack received from [[15019,1],4]
[cn3:21265] [[15019,0],2] connect-ack version from [[15019,1],4] matches ours
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] connect-ack [[15019,1],4] authenticated
[cn3:21265] [[15019,0],2] tcp:peer_accept called for peer [[15019,1],4] in state UNKNOWN on socket 12
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] SEND CONNECT ACK
[cn3:21265] [[15019,0],2] send blocking of 47 bytes to socket 12
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] blocking send complete to socket 12
[cn3:21265] [[15019,0],2]-[[15019,1],4] tcp_peer_connected on socket 12
[cn3:21265] [[15019,0],2] routed_binomial_update: [[15019,1],4] --> [[15019,1],4]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2]-[[15019,1],4] accepted: 192.168.1.3 - 192.168.1.3 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 818 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1193 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] tcp:set_module called for peer [[15019,1],4]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] connection_event_handler: working connection (13, 0) 192.168.1.2:49416
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 743 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] accept_connection: 192.168.1.2:49416
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1165 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:tcp:recv:handler called for peer [[15019,1],4]
[cn3:21265] [[15019,0],2]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:tcp:recv:handler allocate data region of size 61
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 195 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1]:tcp:recv:handler called
[cn2:00814] [[15019,0],1] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 13
[cn2:00814] [[15019,0],1] waiting for connect ack from UNKNOWN
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] connect ack received from UNKNOWN
[cn2:00814] [[15019,0],1] connect-ack recvd from UNKNOWN
[cn2:00814] [[15019,0],1] mca_oob_tcp_recv_connect: connection from new peer
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] connect-ack header from [[15019,1],2] is okay
[cn2:00814] [[15019,0],1] waiting for connect ack from [[15019,1],2]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] connect ack received from [[15019,1],2]
[cn2:00814] [[15019,0],1] connect-ack version from [[15019,1],2] matches ours
[cn2:00814] [[15019,0],1] connect-ack [[15019,1],2] authenticated
[cn2:00814] [[15019,0],1] tcp:peer_accept called for peer [[15019,1],2] in state UNKNOWN on socket 13
[cn2:00814] [[15019,0],1] SEND CONNECT ACK
[cn2:00814] [[15019,0],1] send blocking of 47 bytes to socket 13
[cn2:00814] [[15019,0],1] blocking send complete to socket 13
[cn2:00814] [[15019,0],1]-[[15019,1],2] tcp_peer_connected on socket 13
[cn2:00814] [[15019,0],1] routed_binomial_update: [[15019,1],2] --> [[15019,1],2]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]-[[15019,1],2] accepted: 192.168.1.2 - 192.168.1.2 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 998 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 494 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 683 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] tcp:set_module called for peer [[15019,1],2]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:tcp:recv:handler called for peer [[15019,1],2]
[cn2:00814] [[15019,0],1]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1]:tcp:recv:handler allocate data region of size 61
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 584 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1462 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 267 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 369 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 559 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 738 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 190 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 488 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 88 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 160 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 289 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 442 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 247 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 160 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 69 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1251 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 639 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 109 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1132 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 753 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] connection_event_handler: working connection (13, 11) 192.168.1.3:49702
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] accept_connection: 192.168.1.3:49702
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1406 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 874 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] connection_event_handler: working connection (14, 11) 192.168.1.2:49417
[cn2:00814] [[15019,0],1] accept_connection: 192.168.1.2:49417
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1006 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1481 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 401 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2]:tcp:recv:handler called
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 13
[cn3:21265] [[15019,0],2] waiting for connect ack from UNKNOWN
[cn3:21265] [[15019,0],2] connect ack received from UNKNOWN
[cn3:21265] [[15019,0],2] connect-ack recvd from UNKNOWN
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] mca_oob_tcp_recv_connect: connection from new peer
[cn3:21265] [[15019,0],2] connect-ack header from [[15019,1],5] is okay
[cn3:21265] [[15019,0],2] waiting for connect ack from [[15019,1],5]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 810 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]:tcp:recv:handler called
[cn2:00814] [[15019,0],1] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 14
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] waiting for connect ack from UNKNOWN
[cn2:00814] [[15019,0],1] connect ack received from UNKNOWN
[cn2:00814] [[15019,0],1] connect-ack recvd from UNKNOWN
[cn2:00814] [[15019,0],1] mca_oob_tcp_recv_connect: connection from new peer
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 217 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] connect-ack header from [[15019,1],3] is okay
[cn2:00814] [[15019,0],1] waiting for connect ack from [[15019,1],3]
[cn2:00814] [[15019,0],1] connect ack received from [[15019,1],3]
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] connect-ack version from [[15019,1],3] matches ours
[cn2:00814] [[15019,0],1] connect-ack [[15019,1],3] authenticated
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] connect ack received from [[15019,1],5]
[cn3:21265] [[15019,0],2] connect-ack version from [[15019,1],5] matches ours
[cn3:21265] [[15019,0],2] connect-ack [[15019,1],5] authenticated
[cn2:00814] [[15019,0],1] tcp:peer_accept called for peer [[15019,1],3] in state UNKNOWN on socket 14
[cn2:00814] [[15019,0],1] SEND CONNECT ACK
[cn2:00814] [[15019,0],1] send blocking of 47 bytes to socket 14
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:peer_accept called for peer [[15019,1],5] in state UNKNOWN on socket 13
[cn3:21265] [[15019,0],2] SEND CONNECT ACK
[cn3:21265] [[15019,0],2] send blocking of 47 bytes to socket 13
[cn2:00814] [[15019,0],1] blocking send complete to socket 14
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] blocking send complete to socket 13
[cn2:00814] [[15019,0],1]-[[15019,1],3] tcp_peer_connected on socket 14
[cn2:00814] [[15019,0],1] routed_binomial_update: [[15019,1],3] --> [[15019,1],3]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2]-[[15019,1],5] tcp_peer_connected on socket 13
[cn3:21265] [[15019,0],2] routed_binomial_update: [[15019,1],5] --> [[15019,1],5]
[cn2:00814] [[15019,0],1]-[[15019,1],3] accepted: 192.168.1.2 - 192.168.1.2 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]-[[15019,1],5] accepted: 192.168.1.3 - 192.168.1.3 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn2:00814] [[15019,0],1] tcp:set_module called for peer [[15019,1],3]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 530 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2] tcp:set_module called for peer [[15019,1],5]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:tcp:recv:handler called for peer [[15019,1],3]
[cn2:00814] [[15019,0],1]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn2:00814] [[15019,0],1]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1]:tcp:recv:handler allocate data region of size 61
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2]:tcp:recv:handler called for peer [[15019,1],5]
[cn3:21265] [[15019,0],2]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 517 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2]:tcp:recv:handler allocate data region of size 61
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 618 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 975 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 263 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1179 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 304 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 263 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 304 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 104 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 202 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 443 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1256 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 221 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 276 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 370 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1329 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 221 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 104 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 102 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 374 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 104 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 102 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 374 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 267 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 738 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 190 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 105 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 419 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 121 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1251 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 898 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 109 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 127 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 419 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1251 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] connection_event_handler: working connection (13, 0) 192.168.1.4:39597
[cn4:17677] [[15019,0],3] accept_connection: 192.168.1.4:39597
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 851 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:tcp:recv:handler called
[cn4:17677] [[15019,0],3] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 13
[cn4:17677] [[15019,0],3] waiting for connect ack from UNKNOWN
[cn4:17677] [[15019,0],3] connect ack received from UNKNOWN
[cn4:17677] [[15019,0],3] connect-ack recvd from UNKNOWN
[cn4:17677] [[15019,0],3] mca_oob_tcp_recv_connect: connection from new peer
[cn4:17677] [[15019,0],3] connect-ack header from [[15019,1],6] is okay
[cn4:17677] [[15019,0],3] waiting for connect ack from [[15019,1],6]
[cn4:17677] [[15019,0],3] connect ack received from [[15019,1],6]
[cn4:17677] [[15019,0],3] connect-ack version from [[15019,1],6] matches ours
[cn4:17677] [[15019,0],3] connect-ack [[15019,1],6] authenticated
[cn4:17677] [[15019,0],3] tcp:peer_accept called for peer [[15019,1],6] in state UNKNOWN on socket 13
[cn4:17677] [[15019,0],3] SEND CONNECT ACK
[cn4:17677] [[15019,0],3] send blocking of 47 bytes to socket 13
[cn4:17677] [[15019,0],3] blocking send complete to socket 13
[cn4:17677] [[15019,0],3]-[[15019,1],6] tcp_peer_connected on socket 13
[cn4:17677] [[15019,0],3] routed_binomial_update: [[15019,1],6] --> [[15019,1],6]
[cn4:17677] [[15019,0],3]-[[15019,1],6] accepted: 192.168.1.4 - 192.168.1.4 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1557 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:set_module called for peer [[15019,1],6]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 908 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:tcp:recv:handler called for peer [[15019,1],6]
[cn4:17677] [[15019,0],3]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]:tcp:recv:handler allocate data region of size 61
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1149 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 343 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 109 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 875 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1371 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] connection_event_handler: working connection (14, 11) 192.168.1.4:39598
[cn4:17677] [[15019,0],3] accept_connection: 192.168.1.4:39598
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1002 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3]:tcp:recv:handler called
[cn4:17677] [[15019,0],3] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 14
[cn4:17677] [[15019,0],3] waiting for connect ack from UNKNOWN
[cn4:17677] [[15019,0],3] connect ack received from UNKNOWN
[cn4:17677] [[15019,0],3] connect-ack recvd from UNKNOWN
[cn4:17677] [[15019,0],3] mca_oob_tcp_recv_connect: connection from new peer
[cn4:17677] [[15019,0],3] connect-ack header from [[15019,1],7] is okay
[cn4:17677] [[15019,0],3] waiting for connect ack from [[15019,1],7]
[cn4:17677] [[15019,0],3] connect ack received from [[15019,1],7]
[cn4:17677] [[15019,0],3] connect-ack version from [[15019,1],7] matches ours
[cn4:17677] [[15019,0],3] connect-ack [[15019,1],7] authenticated
[cn4:17677] [[15019,0],3] tcp:peer_accept called for peer [[15019,1],7] in state UNKNOWN on socket 14
[cn4:17677] [[15019,0],3] SEND CONNECT ACK
[cn4:17677] [[15019,0],3] send blocking of 47 bytes to socket 14
[cn4:17677] [[15019,0],3] blocking send complete to socket 14
[cn4:17677] [[15019,0],3]-[[15019,1],7] tcp_peer_connected on socket 14
[cn4:17677] [[15019,0],3] routed_binomial_update: [[15019,1],7] --> [[15019,1],7]
[cn4:17677] [[15019,0],3]-[[15019,1],7] accepted: 192.168.1.4 - 192.168.1.4 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] tcp:set_module called for peer [[15019,1],7]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 195 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:tcp:recv:handler called for peer [[15019,1],7]
[cn4:17677] [[15019,0],3]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]:tcp:recv:handler allocate data region of size 61
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1462 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4]:tcp:recv:handler called for peer [[15019,1],8]
[cn5:14014] [[15019,0],4]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] RECVD COMPLETE MESSAGE FROM [[15019,1],8] (ORIGIN [[15019,1],8]) OF 61 BYTES FOR DEST [[15019,0],4] TAG 1
[cn5:14014] [[15019,0],4] DELIVERING TO RML
[cn5:14014] [[15019,0],4] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] message received 61 bytes from [[15019,1],8] for tag 1
[cn5:14014] [[15019,0],4] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn5:14014] [[15019,0],4] orted_recv: received sync+nidmap from local proc [[15019,1],8]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,1],8] at tag 20
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,1],8] at tag 20
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,1],8]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,1],8]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,1],8]:20
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,1],8]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,1],8]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,1],8]:20
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,1],8]) --> [[15019,1],8]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,1],8] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,1],8]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,1],8]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,1],8]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,1],8] OF 14140 BYTES ON SOCKET 12
[cn5:14014] [[15019,0],4]-[[15019,1],8] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 351 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4]:tcp:recv:handler called for peer [[15019,1],9]
[cn5:14014] [[15019,0],4]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] RECVD COMPLETE MESSAGE FROM [[15019,1],9] (ORIGIN [[15019,1],9]) OF 61 BYTES FOR DEST [[15019,0],4] TAG 1
[cn5:14014] [[15019,0],4] DELIVERING TO RML
[cn5:14014] [[15019,0],4] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] message received 61 bytes from [[15019,1],9] for tag 1
[cn5:14014] [[15019,0],4] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn5:14014] [[15019,0],4] orted_recv: received sync+nidmap from local proc [[15019,1],9]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,1],9] at tag 20
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,1],9] at tag 20
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,1],9]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,1],9]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,1],9]:20
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,1],9]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,1],9]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,1],9]:20
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,1],9]) --> [[15019,1],9]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,1],9] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,1],9]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,1],9]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,1],9]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,1],9] OF 14140 BYTES ON SOCKET 13
[cn5:14014] [[15019,0],4]-[[15019,1],9] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 5
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 5
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:5
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:5
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 179 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 103 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 83 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:tcp:recv:handler called for peer [[15019,1],2]
[cn2:00814] [[15019,0],1]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 95 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 82 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] RECVD COMPLETE MESSAGE FROM [[15019,1],2] (ORIGIN [[15019,1],2]) OF 61 BYTES FOR DEST [[15019,0],1] TAG 1
[cn2:00814] [[15019,0],1] DELIVERING TO RML
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] message received 61 bytes from [[15019,1],2] for tag 1
[cn2:00814] [[15019,0],1] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 108 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] orted_recv: received sync+nidmap from local proc [[15019,1],2]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,1],2] at tag 20
[cn3:21265] [[15019,0],2]:tcp:recv:handler called for peer [[15019,1],4]
[cn3:21265] [[15019,0],2]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,1],2] at tag 20
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,1],2]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,1],2]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,1],2]:20
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,1],2]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,1],2]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,1],2]:20
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,1],2]) --> [[15019,1],2]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,1],2] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,1],2]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,1],2]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,1],2]
[cn3:21265] [[15019,0],2] RECVD COMPLETE MESSAGE FROM [[15019,1],4] (ORIGIN [[15019,1],4]) OF 61 BYTES FOR DEST [[15019,0],2] TAG 1
[cn3:21265] [[15019,0],2] DELIVERING TO RML
[cn3:21265] [[15019,0],2] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,1],2] OF 14140 BYTES ON SOCKET 13
[cn3:21265] [[15019,0],2] message received 61 bytes from [[15019,1],4] for tag 1
[cn3:21265] [[15019,0],2] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn2:00814] [[15019,0],1]-[[15019,1],2] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] orted_recv: received sync+nidmap from local proc [[15019,1],4]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 273 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 108 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,1],4] at tag 20
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,1],4] at tag 20
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,1],4]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,1],4]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,1],4]:20
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,1],4]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,1],4]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,1],4]:20
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,1],4]) --> [[15019,1],4]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,1],4] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,1],4]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,1],4]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,1],4]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,1],4] OF 14140 BYTES ON SOCKET 12
[cn3:21265] [[15019,0],2]-[[15019,1],4] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 273 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 108 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2]:tcp:recv:handler called for peer [[15019,1],5]
[cn3:21265] [[15019,0],2]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] RECVD COMPLETE MESSAGE FROM [[15019,1],5] (ORIGIN [[15019,1],5]) OF 61 BYTES FOR DEST [[15019,0],2] TAG 1
[cn3:21265] [[15019,0],2] DELIVERING TO RML
[cn3:21265] [[15019,0],2] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] message received 61 bytes from [[15019,1],5] for tag 1
[cn3:21265] [[15019,0],2] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn3:21265] [[15019,0],2] orted_recv: received sync+nidmap from local proc [[15019,1],5]
[cn2:00814] [[15019,0],1]:tcp:recv:handler called for peer [[15019,1],3]
[cn2:00814] [[15019,0],1]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] RECVD COMPLETE MESSAGE FROM [[15019,1],3] (ORIGIN [[15019,1],3]) OF 61 BYTES FOR DEST [[15019,0],1] TAG 1
[cn2:00814] [[15019,0],1] DELIVERING TO RML
[cn2:00814] [[15019,0],1] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] message received 61 bytes from [[15019,1],3] for tag 1
[cn2:00814] [[15019,0],1] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn2:00814] [[15019,0],1] orted_recv: received sync+nidmap from local proc [[15019,1],3]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,1],5] at tag 20
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,1],5] at tag 20
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,1],5]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,1],5]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,1],5]:20
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,1],5]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,1],5]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,1],5]:20
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,1],5]) --> [[15019,1],5]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,1],5] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,1],5]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,1],5]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,1],5]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,1],5] OF 14140 BYTES ON SOCKET 13
[cn3:21265] [[15019,0],2]-[[15019,1],5] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 5
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 5
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:5
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:5
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 179 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,1],3] at tag 20
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,1],3] at tag 20
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,1],3]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,1],3]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,1],3]:20
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,1],3]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,1],3]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,1],3]:20
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,1],3]) --> [[15019,1],3]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,1],3] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,1],3]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 103 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,1],3]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,1],3]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,1],3] OF 14140 BYTES ON SOCKET 14
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1]-[[15019,1],3] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 5
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 5
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:5
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:5
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 179 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 278 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 273 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 108 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3]:tcp:recv:handler called for peer [[15019,1],6]
[cn4:17677] [[15019,0],3]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] RECVD COMPLETE MESSAGE FROM [[15019,1],6] (ORIGIN [[15019,1],6]) OF 61 BYTES FOR DEST [[15019,0],3] TAG 1
[cn4:17677] [[15019,0],3] DELIVERING TO RML
[cn4:17677] [[15019,0],3] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] message received 61 bytes from [[15019,1],6] for tag 1
[cn4:17677] [[15019,0],3] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn4:17677] [[15019,0],3] orted_recv: received sync+nidmap from local proc [[15019,1],6]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,1],6] at tag 20
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,1],6] at tag 20
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,1],6]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,1],6]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,1],6]:20
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,1],6]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,1],6]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,1],6]:20
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,1],6]) --> [[15019,1],6]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,1],6] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,1],6]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,1],6]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,1],6]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,1],6] OF 14140 BYTES ON SOCKET 13
[cn4:17677] [[15019,0],3]-[[15019,1],6] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 273 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 108 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3]:tcp:recv:handler called for peer [[15019,1],7]
[cn4:17677] [[15019,0],3]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] RECVD COMPLETE MESSAGE FROM [[15019,1],7] (ORIGIN [[15019,1],7]) OF 61 BYTES FOR DEST [[15019,0],3] TAG 1
[cn4:17677] [[15019,0],3] DELIVERING TO RML
[cn4:17677] [[15019,0],3] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] message received 61 bytes from [[15019,1],7] for tag 1
[cn4:17677] [[15019,0],3] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn4:17677] [[15019,0],3] orted_recv: received sync+nidmap from local proc [[15019,1],7]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,1],7] at tag 20
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,1],7] at tag 20
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,1],7]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,1],7]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,1],7]:20
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,1],7]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,1],7]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,1],7]:20
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,1],7]) --> [[15019,1],7]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,1],7] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,1],7]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,1],7]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,1],7]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,1],7] OF 14140 BYTES ON SOCKET 14
[cn4:17677] [[15019,0],3]-[[15019,1],7] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 5
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 5
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:5
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:5
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 179 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 351 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 608 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 156 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 156 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 103 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 156 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 429 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 482 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 83 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 555 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 429 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 83 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 555 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 397 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 2495 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 83 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 83 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1033 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 2019 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 115 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 608 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 83 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] mca_oob_tcp_listen_thread: new connection: (20, 11) 192.168.1.1:43935
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1875 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] connection_handler: working connection (20, 0) 192.168.1.1:43935
[cn1:14018] [[15019,0],0] accept_connection: 192.168.1.1:43935
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 5
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 5
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 5
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 225 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 5
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],4] for tag 5
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] plm:base:receive processing msg
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] plm:base:receive update proc state command from [[15019,0],4]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn1:14018] [[15019,0],0] plm:base:receive got update_proc_state for job [15019,1]
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] plm:base:receive got update_proc_state for vpid 8 state RUNNING exit_code 0
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] plm:base:receive got update_proc_state for vpid 9 state RUNNING exit_code 0
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] plm:base:receive done processing commands
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],1] for tag 5
[cn1:14018] [[15019,0],0] plm:base:receive processing msg
[cn1:14018] [[15019,0],0] plm:base:receive update proc state command from [[15019,0],1]
[cn1:14018] [[15019,0],0] plm:base:receive got update_proc_state for job [15019,1]
[cn1:14018] [[15019,0],0] plm:base:receive got update_proc_state for vpid 2 state RUNNING exit_code 0
[cn1:14018] [[15019,0],0] plm:base:receive got update_proc_state for vpid 3 state RUNNING exit_code 0
[cn1:14018] [[15019,0],0] plm:base:receive done processing commands
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],3] for tag 5
[cn1:14018] [[15019,0],0] plm:base:receive processing msg
[cn1:14018] [[15019,0],0] plm:base:receive update proc state command from [[15019,0],3]
[cn1:14018] [[15019,0],0] plm:base:receive got update_proc_state for job [15019,1]
[cn1:14018] [[15019,0],0] plm:base:receive got update_proc_state for vpid 6 state RUNNING exit_code 0
[cn1:14018] [[15019,0],0] plm:base:receive got update_proc_state for vpid 7 state RUNNING exit_code 0
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] plm:base:receive done processing commands
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],2] for tag 5
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] plm:base:receive processing msg
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1687 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] plm:base:receive update proc state command from [[15019,0],2]
[cn1:14018] [[15019,0],0] plm:base:receive got update_proc_state for job [15019,1]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] plm:base:receive got update_proc_state for vpid 4 state RUNNING exit_code 0
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] plm:base:receive got update_proc_state for vpid 5 state RUNNING exit_code 0
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] plm:base:receive done processing commands
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 311
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 311 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 311
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 311 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 311
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 311 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called
[cn1:14018] [[15019,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 20
[cn1:14018] [[15019,0],0] waiting for connect ack from UNKNOWN
[cn1:14018] [[15019,0],0] connect ack received from UNKNOWN
[cn1:14018] [[15019,0],0] connect-ack recvd from UNKNOWN
[cn1:14018] [[15019,0],0] mca_oob_tcp_recv_connect: connection from new peer
[cn1:14018] [[15019,0],0] connect-ack header from [[15019,1],0] is okay
[cn1:14018] [[15019,0],0] waiting for connect ack from [[15019,1],0]
[cn1:14018] [[15019,0],0] connect ack received from [[15019,1],0]
[cn1:14018] [[15019,0],0] connect-ack version from [[15019,1],0] matches ours
[cn1:14018] [[15019,0],0] connect-ack [[15019,1],0] authenticated
[cn1:14018] [[15019,0],0] tcp:peer_accept called for peer [[15019,1],0] in state UNKNOWN on socket 20
[cn1:14018] [[15019,0],0] SEND CONNECT ACK
[cn1:14018] [[15019,0],0] send blocking of 47 bytes to socket 20
[cn1:14018] [[15019,0],0] blocking send complete to socket 20
[cn1:14018] [[15019,0],0]-[[15019,1],0] tcp_peer_connected on socket 20
[cn1:14018] [[15019,0],0] routed_binomial_update: [[15019,1],0] --> [[15019,1],0]
[cn1:14018] [[15019,0],0]-[[15019,1],0] accepted: 192.168.1.1 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:14033] progress: initialized event flag to: 3
[cn1:14033] progress: initialized yield_when_idle to: false
[cn1:14033] progress: initialized num users to: 0
[cn1:14033] progress: initialized poll rate to: 23940000
[cn1:14033] progress: event_users_increment setting count to 1
[cn1:14033] mca: base: components_register: registering ess components
[cn1:14033] mca: base: components_register: found loaded component env
[cn1:14033] mca: base: components_register: component env has no register or open function
[cn1:14033] mca: base: components_open: opening ess components
[cn1:14033] mca: base: components_open: found loaded component env
[cn1:14033] mca: base: components_open: component env open function successful
[cn1:14033] mca:base:select: Auto-selecting ess components
[cn1:14033] mca:base:select:(  ess) Querying component [env]
[cn1:14033] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn1:14033] mca:base:select:(  ess) Selected component [env]
[cn1:14033] ess:env set name to [[15019,1],0]
[cn1:14033] mca: base: components_register: registering oob components
[cn1:14033] mca: base: components_register: found loaded component tcp
[cn1:14033] mca: base: components_register: component tcp register function successful
[cn1:14033] mca: base: components_open: opening oob components
[cn1:14033] mca: base: components_open: found loaded component tcp
[cn1:14033] mca: base: components_open: component tcp open function successful
[cn1:14033] mca:oob:select: checking available component tcp
[cn1:14033] mca:oob:select: Querying component [tcp]
[cn1:14033] oob:tcp: component_available called
[cn1:14033] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn1:14033] [[15019,1],0] oob:tcp:init rejecting loopback interface lo
[cn1:14033] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn1:14033] [[15019,1],0] oob:tcp:init creating module for V4 address on interface eth0
[cn1:14033] [[15019,1],0] creating OOB-TCP module for interface eth0
[cn1:14033] [[15019,1],0] oob:tcp:init adding 192.168.1.1 to our list of V4 connections
[cn1:14033] WORKING INTERFACE 3 KERNEL INDEX 3 FAMILY: V4
[cn1:14033] [[15019,1],0] oob:tcp:init creating module for V4 address on interface eth1
[cn1:14033] [[15019,1],0] creating OOB-TCP module for interface eth1
[cn1:14033] [[15019,1],0] oob:tcp:init adding 192.168.10.2 to our list of V4 connections
[cn1:14033] [[15019,1],0] TCP STARTUP
[cn1:14033] [[15019,1],0] attempting to bind to IPv4 port 0
[cn1:14033] [[15019,1],0] assigned IPv4 port 56591
[cn1:14033] mca:oob:select: Adding component to end
[cn1:14033] mca:oob:select: Found 1 active transports
[cn1:14033] mca: base: components_register: registering rml components
[cn1:14033] mca: base: components_register: found loaded component oob
[cn1:14033] mca: base: components_register: component oob has no register or open function
[cn1:14033] mca: base: components_open: opening rml components
[cn1:14033] mca: base: components_open: found loaded component oob
[cn1:14033] mca: base: components_open: component oob open function successful
[cn1:14033] orte_rml_base_select: initializing rml component oob
[cn1:14033] mca: base: components_register: registering routed components
[cn1:14033] mca: base: components_register: found loaded component binomial
[cn1:14033] mca: base: components_register: component binomial has no register or open function
[cn1:14033] mca: base: components_register: found loaded component debruijn
[cn1:14033] mca: base: components_register: component debruijn has no register or open function
[cn1:14033] mca: base: components_register: found loaded component direct
[cn1:14033] mca: base: components_register: component direct has no register or open function
[cn1:14033] mca: base: components_register: found loaded component radix
[cn1:14033] mca: base: components_register: component radix register function successful
[cn1:14033] mca: base: components_open: opening routed components
[cn1:14033] mca: base: components_open: found loaded component binomial
[cn1:14033] mca: base: components_open: found loaded component debruijn
[cn1:14033[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 311
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 311 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 311 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 311 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 311 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0] tcp:set_module called for peer [[15019,1],0]
[cn1:14018] [[15019,0],0] message received 311 bytes from [[15019,0],4] for tag 2
] mca: base: components_open: found loaded component direct
[cn1:14033] mca: base: components_open: found loaded component radix
[cn1:14033] mca:base:select: Auto-selecting routed components
[cn1:14033] mca:base:select:(routed) Querying component [binomial]
[cn1:14033] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn1:14033] mca:base:select:(routed) Querying component [debruijn]
[cn1:14033] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn1:14033] mca:base:select:(routed) Querying component [direct]
[cn1:14033] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn1:14033] mca:base:select:(routed) Querying component [radix]
[cn1:14033] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn1:14033] mca:base:select:(routed) Selected component [binomial]
[cn1:14033] mca: base: close: unloading component debruijn
[cn1:14033] mca: base: close: unloading component direct
[cn1:14033] mca: base: close: unloading component radix
[cn1:14033] orte_routed_base_select: initializing selected component binomial
[cn1:14033] mca: base: components_register: registering dstore components
[cn1:14033] mca: base: components_register: found loaded component hash
[cn1:14033] mca: base: components_register: component hash register function successful
[cn1:14033] mca: base: components_open: opening dstore components
[cn1:14033] mca: base: components_open: found loaded component hash
[cn1:14033] mca:dstore:select: checking available component hash
[cn1:14033] mca:dstore:select: Querying component [hash]
[cn1:14033] mca: base: components_register: registering grpcomm components
[cn1:14033] mca: base: components_register: found loaded component bad
[cn1:14033] mca: base: components_register: component bad has no register or open function
[cn1:14033] mca: base: components_open: opening grpcomm components
[cn1:14033] mca: base: components_open: found loaded component bad
[cn1:14033] mca: base: components_open: component bad open function successful
[cn1:14033] mca:base:select: Auto-selecting grpcomm components
[cn1:14033] mca:base:select:(grpcomm) Querying component [bad]
[cn1:14033] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn1:14033] mca:base:select:(grpcomm) Selected component [bad]
[cn1:14033] [[15019,1],0] grpcomm:base:receive start comm
[cn1:14033] [[15019,1],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn1:14033] [[15019,1],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn1:14033] [[15019,1],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn1:14033] [[15019,1],0] setting up session dir with
	tmpdir: /tmp
	host cn1
[cn1:14033] [[15019,1],0] posting recv
[cn1:14033] [[15019,1],0] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn1:14033] [[15019,1],0] posting recv
[cn1:14033] [[15019,1],0] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn1:14033] [[15019,1],0] posting recv
[cn1:14033] [[15019,1],0] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn1:14033] procdir: /tmp/openmpi-sessions-artpol@cn1_0/15019/1/0
[cn1:14033] jobdir: /tmp/openmpi-sessions-artpol@cn1_0/15019/1
[cn1:14033] top: openmpi-sessions-artpol@cn1_0
[cn1:14033] tmp: /tmp
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 984285185
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 984285185
[cn1:14033] [[15019,1],0] routed_binomial: init routes for proc job [15019,1]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
	daemon uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn1:14033] [[15019,1],0] [[15019,0],0] with daemon registering
[cn1:14033] [[15019,1],0] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn1:14033] [[15019,1],0] rml_send_buffer to peer [[15019,0[cn2:00817] progress: initialized event flag to: 3
[cn2:00817] progress: initialized yield_when_idle to: false
[cn2:00817] progress: initialized num users to: 0
[cn2:00817] progress: initialized poll rate to: 23940000
[cn2:00817] progress: event_users_increment setting count to 1
[cn4:17680] progress: initialized event flag to: 3
[cn4:17680] progress: initialized yield_when_idle to: false
[cn4:17680] progress: initialized num users to: 0
[cn4:17680] progress: initialized poll rate to: 23940000
[cn4:17680] progress: event_users_increment setting count to 1
[cn3:21268] progress: initialized event flag to: 3
[cn3:21268] progress: initialized yield_when_idle to: false
[cn3:21268] progress: initialized num users to: 0
[cn3:21268] progress: initialized poll rate to: 23940000
[cn3:21268] progress: event_users_increment setting count to 1
[cn5:14017] progress: initialized event flag to: 3
[cn5:14017] progress: initialized yield_when_idle to: false
[cn5:14017] progress: initialized num users to: 0
[cn5:14017] progress: initialized poll rate to: 23940000
[cn5:14017] progress: event_users_increment setting count to 1
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 311
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 311 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 81
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 81 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 311
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 311 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 311
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 311 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 311 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 81 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 311 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 311 bytes from [[15019,0],2] for tag 2
],0] at tag 1
[cn1:14033] [[15019,1],0] registering sync waiting for ack
[cn1:14033] [[15019,1],0] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn1:14033] [[15019,1],0]: set_addr to uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn1:14033] [[15019,1],0]:set_addr checking if peer [[15019,0],0] is reachable via component tcp
[cn1:14033] [[15019,1],0] oob:tcp: working peer [[15019,0],0] address tcp://192.168.1.1,192.168.10.2:57056
[cn1:14033] [[15019,1],0] PEER [[15019,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:14033] [[15019,1],0] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:14033] [[15019,1],0]:tcp set addr for peer [[15019,0],0]
[cn1:14033] [[15019,1],0] PEER [[15019,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 3 INTERFACE eth1
[cn1:14033] [[15019,1],0] PASSING ADDR 192.168.10.2 TO INTERFACE eth1 AT KERNEL INDEX 3
[cn1:14033] [[15019,1],0]:tcp set addr for peer [[15019,0],0]
[cn1:14033] [[15019,1],0]: peer [[15019,0],0] is reachable via component tcp
[cn1:14033] [[15019,1],0] posting recv
[cn1:14033] [[15019,1],0] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn1:14033] [[15019,1],0] rml_send_msg to peer [[15019,0],0] at tag 1
[cn1:14033] [[15019,1],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14033] [[15019,1],0]:tcp:processing set_peer cmd for interface eth0
[cn1:14033] [[15019,1],0] SET_PEER ADDING PEER [[15019,0],0]
[cn1:14033] [[15019,1],0] set_peer: peer [[15019,0],0] is listening on interface eth0 net 192.168.1.1 port 57056
[cn1:14033] [[15019,1],0]:tcp:processing set_peer cmd for interface eth1
[cn1:14033] [[15019,1],0] SET_PEER ADDING PEER [[15019,0],0]
[cn1:14033] [[15019,1],0] set_peer: peer [[15019,0],0] is listening on interface eth1 net 192.168.10.2 port 57056
[cn1:14033] [[15019,1],0] oob:base:send to target [[15019,0],0]
[cn1:14033] [[15019,1],0] oob:tcp:send_nb to peer [[15019,0],0]:1
[cn1:14033] [[15019,1],0] tcp:send_nb to peer [[15019,0],0]
[cn1:14033] [[15019,1],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14033] [[15019,1],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:1
[cn1:14033] [[15019,1],0] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14033] [[15019,1],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],0]
[cn1:14033] [[15019,1],0] tcp:send_nb: initiating connection to [[15019,0],0]
[cn1:14033] [[15019,1],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],0]
[cn1:14033] [[15019,1],0] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] via interface eth0
[cn1:14033] [[15019,1],0] oob:tcp:peer creating socket to [[15019,0],0]
[cn1:14033] [[15019,1],0] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] via interface eth0 on socket 11
[cn1:14033] [[15019,1],0] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] on 192.168.1.1:57056 - 0 retries
[cn1:14033] [[15019,1],0] waiting for connect completion to [[15019,0],0] - activating send event
[cn1:14033] [[15019,1],0] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14033] [[15019,1],0] tcp:send_handler CONNECTING
[cn1:14033] [[15019,1],0]:tcp:complete_connect called for peer [[15019,0],0] on socket 11
[cn1:14033] [[15019,1],0] tcp_peer_complete_connect: sending ack to [[15019,0],0]
[cn1:14033] [[15019,1],0] SEND CONNECT ACK
[cn1:14033] fetching data from INTERNAL dstore
[cn1:14033] dstore:hash:fetch: searching for key opal.cred on proc 984285185
[cn1:14033] dstore_hash:fetch key opal.cred for proc 984285185 not found
[cn1:14033] [[15019,1],0] send blocking of 47 bytes to socket 11
[cn1:14033] [[15019,1],0] blocking send complete to socket 11
[cn1:14033] [[15019,1],0] tcp_peer_complete_connect: [cn5:14018] progress: initialized event flag to: 3
[cn5:14018] progress: initialized yield_when_idle to: false
[cn5:14018] progress: initialized num users to: 0
[cn5:14018] progress: initialized poll rate to: 23940000
[cn5:14018] progress: event_users_increment setting count to 1
[cn2:00818] progress: initialized event flag to: 3
[cn4:17681] progress: initialized event flag to: 3
[cn4:17681] progress: initialized yield_when_idle to: false
[cn4:17681] progress: initialized num users to: 0
[cn4:17681] progress: initialized poll rate to: 23940000
[cn4:17681] progress: event_users_increment setting count to 1
[cn3:21269] progress: initialized event flag to: 3
[cn3:21269] progress: initialized yield_when_idle to: false
[cn3:21269] progress: initialized num users to: 0
[cn3:21269] progress: initialized poll rate to: 23940000
[cn3:21269] progress: event_users_increment setting count to 1
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 779
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 779 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 260
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 260 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 779
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 779 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 779 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 260 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 779 bytes from [[15019,0],2] for tag 2
setting read event on connection to [[15019,0],0]
[cn5:14017] mca: base: components_register: registering ess components
[cn5:14017] mca: base: components_register: found loaded component env
[cn5:14017] mca: base: components_register: component env has no register or open function
[cn5:14017] mca: base: components_open: opening ess components
[cn5:14017] mca: base: components_open: found loaded component env
[cn5:14017] mca: base: components_open: component env open function successful
[cn5:14017] mca:base:select: Auto-selecting ess components
[cn5:14017] mca:base:select:(  ess) Querying component [env]
[cn5:14017] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn5:14017] mca:base:select:(  ess) Selected component [env]
[cn5:14017] ess:env set name to [[15019,1],8]
[cn2:00818] progress: initialized yield_when_idle to: false
[cn2:00818] progress: initialized num users to: 0
[cn2:00818] progress: initialized poll rate to: 23940000
[cn2:00818] progress: event_users_increment setting count to 1
[cn4:17680] mca: base: components_register: registering ess components
[cn3:21268] mca: base: components_register: registering ess components
[cn3:21268] mca: base: components_register: found loaded component env
[cn3:21268] mca: base: components_register: component env has no register or open function
[cn3:21268] mca: base: components_open: opening ess components
[cn3:21268] mca: base: components_open: found loaded component env
[cn3:21268] mca: base: components_open: component env open function successful
[cn3:21268] mca:base:select: Auto-selecting ess components
[cn3:21268] mca:base:select:(  ess) Querying component [env]
[cn3:21268] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn3:21268] mca:base:select:(  ess) Selected component [env]
[cn3:21268] ess:env set name to [[15019,1],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 779
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 779 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 263
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 263 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 779 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 263 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],2] for tag 2
[cn5:14018] mca: base: components_register: registering ess components
[cn5:14018] mca: base: components_register: found loaded component env
[cn5:14018] mca: base: components_register: component env has no register or open function
[cn5:14018] mca: base: components_open: opening ess components
[cn5:14018] mca: base: components_open: found loaded component env
[cn5:14018] mca: base: components_open: component env open function successful
[cn5:14018] mca:base:select: Auto-selecting ess components
[cn5:14018] mca:base:select:(  ess) Querying component [env]
[cn5:14018] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn5:14018] mca:base:select:(  ess) Selected component [env]
[cn5:14018] ess:env set name to [[15019,1],9]
[cn2:00817] mca: base: components_register: registering ess components
[cn2:00817] mca: base: components_register: found loaded component env
[cn2:00817] mca: base: components_register: component env has no register or open function
[cn4:17680] mca: base: components_register: found loaded component env
[cn3:21269] mca: base: components_register: registering ess components
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 500
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 500 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 121
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 121 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 192
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 192 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 500 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 121 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 192 bytes from [[15019,0],2] for tag 2
[cn5:14017] mca: base: components_register: registering oob components
[cn2:00817] mca: base: components_open: opening ess components
[cn2:00817] mca: base: components_open: found loaded component env
[cn2:00817] mca: base: components_open: component env open function successful
[cn2:00817] mca:base:select: Auto-selecting ess components
[cn2:00817] mca:base:select:(  ess) Querying component [env]
[cn2:00817] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn2:00817] mca:base:select:(  ess) Selected component [env]
[cn4:17680] mca: base: components_register: component env has no register or open function
[cn3:21269] mca: base: components_register: found loaded component env
[cn3:21269] mca: base: components_register: component env has no register or open function
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 76
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 76 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 439
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 439 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 500
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 500 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 76 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 439 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 500 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 76
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 76 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] mca: base: components_register: found loaded component tcp
[cn2:00817] ess:env set name to [[15019,1],2]
[cn4:17680] mca: base: components_open: opening ess components
[cn4:17680] mca: base: components_open: found loaded component env
[cn4:17680] mca: base: components_open: component env open function successful
[cn4:17680] mca:base:select: Auto-selecting ess components
[cn4:17680] mca:base:select:(  ess) Querying component [env]
[cn4:17680] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn3:21269] mca: base: components_open: opening ess components
[cn3:21269] mca: base: components_open: found loaded component env
[cn3:21269] mca: base: components_open: component env open function successful
[cn3:21269] mca:base:select: Auto-selecting ess components
[cn3:21269] mca:base:select:(  ess) Querying component [env]
[cn3:21269] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn3:21269] mca:base:select:(  ess) Selected component [env]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 440
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 440 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 137
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 137 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 76 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0] message received 440 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 137 bytes from [[15019,0],3] for tag 2
[cn3:21269] ess:env set name to [[15019,1],5]
[cn5:14017] mca: base: components_register: component tcp register function successful
[cn5:14017] mca: base: components_open: opening oob components
[cn5:14017] mca: base: components_open: found loaded component tcp
[cn5:14017] mca: base: components_open: component tcp open function successful
[cn5:14017] mca:oob:select: checking available component tcp
[cn5:14017] mca:oob:select: Querying component [tcp]
[cn2:00818] mca: base: components_register: registering ess components
[cn4:17680] mca:base:select:(  ess) Selected component [env]
[cn4:17680] ess:env set name to [[15019,1],6]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 78
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 78 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 172
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 172 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 78 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 172 bytes from [[15019,0],2] for tag 2
[cn5:14017] oob:tcp: component_available called
[cn2:00818] mca: base: components_register: found loaded component env
[cn4:17681] mca: base: components_register: registering ess components
[cn3:21268] mca: base: components_register: registering oob components
[cn3:21268] mca: base: components_register: found loaded component tcp
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 500
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 500 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 121
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 121 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 662
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 662 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 500 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 121 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 662 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 247
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 247 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 500
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 500 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 76
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 76 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 401
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 401 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14017] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn5:14017] [[15019,1],8] oob:tcp:init rejecting loopback interface lo
[cn5:14017] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn5:14017] [[15019,1],8] oob:tcp:init creating module for V4 address on interface eth0
[cn5:14017] [[15019,1],8] creating OOB-TCP module for interface eth0
[cn5:14017] [[15019,1],8] oob:tcp:init adding 192.168.1.5 to our list of V4 connections
[cn5:14017] [[15019,1],8] TCP STARTUP
[cn2:00818] mca: base: components_register: component env has no register or open function
[cn4:17681] mca: base: components_register: found loaded component env
[cn4:17681] mca: base: components_register: component env has no register or open function
[cn4:17681] mca: base: components_open: opening ess components
[cn4:17681] mca: base: components_open: found loaded component env
[cn4:17681] mca: base: components_open: component env open function successful
[cn4:17681] mca:base:select: Auto-selecting ess components
[cn4:17681] mca:base:select:(  ess) Querying component [env]
[cn4:17681] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn4:17681] mca:base:select:(  ess) Selected component [env]
[cn3:21268] mca: base: components_register: component tcp register function successful
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 247 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 500 bytes from [[15019,0],1] for tag 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn1:14018] [[15019,0],0] message received 76 bytes from [[15019,0],3] for tag 2
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 401 bytes from [[15019,0],2] for tag 2
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 76
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 76 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 717
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 717 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] [[15019,1],8] attempting to bind to IPv4 port 0
[cn5:14017] [[15019,1],8] assigned IPv4 port 52061
[cn5:14017] mca:oob:select: Adding component to end
[cn5:14017] mca:oob:select: Found 1 active transports
[cn2:00818] mca: base: components_open: opening ess components
[cn2:00818] mca: base: components_open: found loaded component env
[cn2:00818] mca: base: components_open: component env open function successful
[cn2:00818] mca:base:select: Auto-selecting ess components
[cn2:00818] mca:base:select:(  ess) Querying component [env]
[cn2:00818] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn2:00818] mca:base:select:(  ess) Selected component [env]
[cn4:17681] ess:env set name to [[15019,1],7]
[cn3:21268] mca: base: components_open: opening oob components
[cn3:21268] mca: base: components_open: found loaded component tcp
[cn3:21268] mca: base: components_open: component tcp open function successful
[cn3:21268] mca:oob:select: checking available component tcp
[cn3:21268] mca:oob:select: Querying component [tcp]
[cn3:21268] oob:tcp: component_available called
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 76 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 717 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 537
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 537 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] mca: base: components_register: registering rml components
[cn2:00818] ess:env set name to [[15019,1],3]
[cn4:17680] mca: base: components_register: registering oob components
[cn3:21268] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn3:21268] [[15019,1],4] oob:tcp:init rejecting loopback interface lo
[cn3:21268] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn3:21268] [[15019,1],4] oob:tcp:init creating module for V4 address on interface eth0
[cn3:21268] [[15019,1],4] creating OOB-TCP module for interface eth0
[cn3:21268] [[15019,1],4] oob:tcp:init adding 192.168.1.3 to our list of V4 connections
[cn3:21268] [[15019,1],4] TCP STARTUP
[cn3:21268] [[15019,1],4] attempting to bind to IPv4 port 0
[cn3:21268] [[15019,1],4] assigned IPv4 port 53220
[cn3:21268] mca:oob:select: Adding component to end
[cn3:21268] mca:oob:select: Found 1 active transports
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 537 bytes from [[15019,0],2] for tag 2
[cn5:14017] mca: base: components_register: found loaded component oob
[cn2:00817] mca: base: components_register: registering oob components
[cn4:17680] mca: base: components_register: found loaded component tcp
[cn3:21268] mca: base: components_register: registering rml components
[cn3:21268] mca: base: components_register: found loaded component oob
[cn3:21268] mca: base: components_register: component oob has no register or open function
[cn3:21268] mca: base: components_open: opening rml components
[cn3:21268] mca: base: components_open: found loaded component oob
[cn3:21268] mca: base: components_open: component oob open function successful
[cn3:21268] orte_rml_base_select: initializing rml component oob
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 395
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 395 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 488
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 488 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 172
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 172 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 395 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 488 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 172 bytes from [[15019,0],2] for tag 2
[cn5:14017] mca: base: components_register: component oob has no register or open function
[cn5:14017] mca: base: components_open: opening rml components
[cn5:14017] mca: base: components_open: found loaded component oob
[cn5:14017] mca: base: components_open: component oob open function successful
[cn5:14017] orte_rml_base_select: initializing rml component oob
[cn2:00817] mca: base: components_register: found loaded component tcp
[cn4:17680] mca: base: components_register: component tcp register function successful
[cn4:17680] mca: base: components_open: opening oob components
[cn4:17680] mca: base: components_open: found loaded component tcp
[cn4:17680] mca: base: components_open: component tcp open function successful
[cn4:17680] mca:oob:select: checking available component tcp
[cn4:17680] mca:oob:select: Querying component [tcp]
[cn4:17680] oob:tcp: component_available called
[cn3:21269] mca: base: components_register: registering oob components
[cn3:21269] mca: base: components_register: found loaded component tcp
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 172
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 172 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 88
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 88 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 172 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 88 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 401
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 401 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14018] mca: base: components_register: registering oob components
[cn5:14018] mca: base: components_register: found loaded component tcp
[cn2:00817] mca: base: components_register: component tcp register function successful
[cn4:17680] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn3:21269] mca: base: components_register: component tcp register function successful
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 401
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 401 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 442
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 442 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 2433 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 401 bytes from [[15019,0],2] for tag 2
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],4] for tag 2
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 401 bytes from [[15019,0],1] for tag 2
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 442 bytes from [[15019,0],3] for tag 2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 401
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 401 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 88
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 88 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 247
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 247 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 717
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 717 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21269] mca: base: components_open: opening oob components
[cn3:21269] mca: base: components_open: found loaded component tcp
[cn3:21269] mca: base: components_open: component tcp open function successful
[cn3:21269] mca:oob:select: checking available component tcp
[cn3:21269] mca:oob:select: Querying component [tcp]
[cn3:21269] oob:tcp: component_available called
[cn5:14018] mca: base: components_register: component tcp register function successful
[cn2:00817] mca: base: components_open: opening oob components
[cn2:00817] mca: base: components_open: found loaded component tcp
[cn2:00817] mca: base: components_open: component tcp open function successful
[cn2:00817] mca:oob:select: checking available component tcp
[cn2:00817] mca:oob:select: Querying component [tcp]
[cn2:00817] oob:tcp: component_available called
[cn4:17680] [[15019,1],6] oob:tcp:init rejecting loopback interface lo
[cn4:17680] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn4:17680] [[15019,1],6] oob:tcp:init creating module for V4 address on interface eth0
[cn4:17680] [[15019,1],6] creating OOB-TCP module for interface eth0
[cn4:17680] [[15019,1],6] oob:tcp:init adding 192.168.1.4 to our list of V4 connections
[cn4:17680] [[15019,1],6] TCP STARTUP
[cn1:14018] [[15019,0],0] message received 401 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 88 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 247 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 717 bytes from [[15019,0],2] for tag 2
[cn5:14018] mca: base: components_open: opening oob components
[cn5:14018] mca: base: components_open: found loaded component tcp
[cn5:14018] mca: base: components_open: component tcp open function successful
[cn5:14018] mca:oob:select: checking available component tcp
[cn5:14018] mca:oob:select: Querying component [tcp]
[cn5:14018] oob:tcp: component_available called
[cn2:00817] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn4:17680] [[15019,1],6] attempting to bind to IPv4 port 0
[cn4:17680] [[15019,1],6] assigned IPv4 port 51021
[cn4:17680] mca:oob:select: Adding component to end
[cn4:17680] mca:oob:select: Found 1 active transports
[cn3:21269] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn3:21269] [[15019,1],5] oob:tcp:init rejecting loopback interface lo
[cn3:21269] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn3:21269] [[15019,1],5] oob:tcp:init creating module for V4 address on interface eth0
[cn3:21269] [[15019,1],5] creating OOB-TCP module for interface eth0
[cn3:21269] [[15019,1],5] oob:tcp:init adding 192.168.1.3 to our list of V4 connections
[cn3:21269] [[15019,1],5] TCP STARTUP
[cn3:21269] [[15019,1],5] attempting to bind to IPv4 port 0
[cn3:21269] [[15019,1],5] assigned IPv4 port 48732
[cn3:21269] mca:oob:select: Adding component to end
[cn3:21269] mca:oob:select: Found 1 active transports
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 88
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 88 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 172
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 172 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 537
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 537 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 88 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 172 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 537 bytes from [[15019,0],2] for tag 2
[cn5:14018] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn2:00818] mca: base: components_register: registering oob components
[cn2:00818] mca: base: components_register: found loaded component tcp
[cn4:17681] mca: base: components_register: registering oob components
[cn3:21269] mca: base: components_register: registering rml components
[cn3:21269] mca: base: components_register: found loaded component oob
[cn3:21269] mca: base: components_register: component oob has no register or open function
[cn3:21269] mca: base: components_open: opening rml components
[cn3:21269] mca: base: components_open: found loaded component oob
[cn3:21269] mca: base: components_open: component oob open function successful
[cn3:21269] orte_rml_base_select: initializing rml component oob
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 659
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 659 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 276
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 276 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 659 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 276 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 263
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 263 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 401
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 401 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 370
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 370 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14018] [[15019,1],9] oob:tcp:init rejecting loopback interface lo
[cn5:14018] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn5:14018] [[15019,1],9] oob:tcp:init creating module for V4 address on interface eth0
[cn5:14018] [[15019,1],9] creating OOB-TCP module for interface eth0
[cn5:14018] [[15019,1],9] oob:tcp:init adding 192.168.1.5 to our list of V4 connections
[cn5:14018] [[15019,1],9] TCP STARTUP
[cn5:14018] [[15019,1],9] attempting to bind to IPv4 port 0
[cn5:14018] [[15019,1],9] assigned IPv4 port 42002
[cn5:14018] mca:oob:select: Adding component to end
[cn5:14018] mca:oob:select: Found 1 active transports
[cn2:00818] mca: base: components_register: component tcp register function successful
[cn4:17681] mca: base: components_register: found loaded component tcp
[cn3:21268] mca: base: components_register: registering routed components
[cn3:21268] mca: base: components_register: found loaded component binomial
[cn3:21268] mca: base: components_register: component binomial has no register or open function
[cn1:14018] [[15019,0],0] message received 263 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 401 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 370 bytes from [[15019,0],2] for tag 2
[cn5:14018] mca: base: components_register: registering rml components
[cn5:14018] mca: base: components_register: found loaded component oob
[cn5:14018] mca: base: components_register: component oob has no register or open function
[cn2:00818] mca: base: components_open: opening oob components
[cn2:00818] mca: base: components_open: found loaded component tcp
[cn2:00818] mca: base: components_open: component tcp open function successful
[cn2:00818] mca:oob:select: checking available component tcp
[cn2:00818] mca:oob:select: Querying component [tcp]
[cn2:00818] oob:tcp: component_available called
[cn4:17681] mca: base: components_register: component tcp register function successful
[cn3:21268] mca: base: components_register: found loaded component debruijn
[cn3:21268] mca: base: components_register: component debruijn has no register or open function
[cn3:21268] mca: base: components_register: found loaded component direct
[cn3:21268] mca: base: components_register: component direct has no register or open function
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 304
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 304 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 305
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 305 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 401
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 401 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1329
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1329 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 304 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 305 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 401 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 1329 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 276
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 276 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 442
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 442 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 217
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 217 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 221
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 221 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14018] mca: base: components_open: opening rml components
[cn5:14018] mca: base: components_open: found loaded component oob
[cn5:14018] mca: base: components_open: component oob open function successful
[cn5:14018] orte_rml_base_select: initializing rml component oob
[cn2:00818] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn2:00818] [[15019,1],3] oob:tcp:init rejecting loopback interface lo
[cn2:00818] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn2:00818] [[15019,1],3] oob:tcp:init creating module for V4 address on interface eth0
[cn4:17681] mca: base: components_open: opening oob components
[cn4:17681] mca: base: components_open: found loaded component tcp
[cn4:17681] mca: base: components_open: component tcp open function successful
[cn4:17681] mca:oob:select: checking available component tcp
[cn4:17681] mca:oob:select: Querying component [tcp]
[cn4:17681] oob:tcp: component_available called
[cn3:21268] mca: base: components_register: found loaded component radix
[cn3:21268] mca: base: components_register: component radix register function successful
[cn3:21268] mca: base: components_open: opening routed components
[cn3:21268] mca: base: components_open: found loaded component binomial
[cn3:21268] mca: base: components_open: found loaded component debruijn
[cn3:21268] mca: base: components_open: found loaded component direct
[cn3:21268] mca: base: components_open: found loaded component radix
[cn3:21268] mca:base:select: Auto-selecting routed components
[cn3:21268] mca:base:select:(routed) Querying component [binomial]
[cn3:21268] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn3:21268] mca:base:select:(routed) Querying component [debruijn]
[cn3:21268] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn3:21268] mca:base:select:(routed) Querying component [direct]
[cn3:21268] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn3:21268] mca:base:select:(routed) Querying component [radix]
[cn3:21268] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn3:21268] mca:base:select:(routed) Selected component [binomial]
[cn3:21268] mca: base: close: unloading component debruijn
[cn1:14018] [[15019,0],0] message received 276 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 442 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 217 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 221 bytes from [[15019,0],2] for tag 2
[cn5:14017] mca: base: components_register: registering routed components
[cn5:14017] mca: base: components_register: found loaded component binomial
[cn5:14017] mca: base: components_register: component binomial has no register or open function
[cn2:00818] [[15019,1],3] creating OOB-TCP module for interface eth0
[cn2:00818] [[15019,1],3] oob:tcp:init adding 192.168.1.2 to our list of V4 connections
[cn2:00818] [[15019,1],3] TCP STARTUP
[cn2:00818] [[15019,1],3] attempting to bind to IPv4 port 0
[cn2:00818] [[15019,1],3] assigned IPv4 port 45161
[cn2:00818] mca:oob:select: Adding component to end
[cn2:00818] mca:oob:select: Found 1 active transports
[cn4:17681] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn4:17681] [[15019,1],7] oob:tcp:init rejecting loopback interface lo
[cn4:17681] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn3:21268] mca: base: close: unloading component direct
[cn3:21268] mca: base: close: unloading component radix
[cn3:21268] orte_routed_base_select: initializing selected component binomial
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 370
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 370 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 247
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 247 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 530
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 530 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 520
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 520 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 370 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 247 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 530 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 520 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 263
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 263 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 105
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 105 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] mca: base: components_register: found loaded component debruijn
[cn5:14017] mca: base: components_register: component debruijn has no register or open function
[cn5:14017] mca: base: components_register: found loaded component direct
[cn5:14017] mca: base: components_register: component direct has no register or open function
[cn2:00817] [[15019,1],2] oob:tcp:init rejecting loopback interface lo
[cn2:00817] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn2:00817] [[15019,1],2] oob:tcp:init creating module for V4 address on interface eth0
[cn4:17681] [[15019,1],7] oob:tcp:init creating module for V4 address on interface eth0
[cn4:17681] [[15019,1],7] creating OOB-TCP module for interface eth0
[cn4:17681] [[15019,1],7] oob:tcp:init adding 192.168.1.4 to our list of V4 connections
[cn4:17681] [[15019,1],7] TCP STARTUP
[cn4:17681] [[15019,1],7] attempting to bind to IPv4 port 0
[cn4:17681] [[15019,1],7] assigned IPv4 port 49407
[cn4:17681] mca:oob:select: Adding component to end
[cn4:17681] mca:oob:select: Found 1 active transports
[cn3:21268] mca: base: components_register: registering dstore components
[cn3:21268] mca: base: components_register: found loaded component hash
[cn3:21268] mca: base: components_register: component hash register function successful
[cn3:21268] mca: base: components_open: opening dstore components
[cn3:21268] mca: base: components_open: found loaded component hash
[cn3:21268] mca:dstore:select: checking available component hash
[cn3:21268] mca:dstore:select: Querying component [hash]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 670
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 670 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 442
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 442 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 263 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 105 bytes from [[15019,0],2] for tag 2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] message received 670 bytes from [[15019,0],4] for tag 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] message received 442 bytes from [[15019,0],1] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17680] mca: base: components_register: registering rml components
[cn4:17680] mca: base: components_register: found loaded component oob
[cn4:17680] mca: base: components_register: component oob has no register or open function
[cn3:21268] mca: base: components_register: registering grpcomm components
[cn5:14017] mca: base: components_register: found loaded component radix
[cn5:14017] mca: base: components_register: component radix register function successful
[cn5:14017] mca: base: components_open: opening routed components
[cn5:14017] mca: base: components_open: found loaded component binomial
[cn5:14017] mca: base: components_open: found loaded component debruijn
[cn5:14017] mca: base: components_open: found loaded component direct
[cn5:14017] mca: base: components_open: found loaded component radix
[cn5:14017] mca:base:select: Auto-selecting routed components
[cn5:14017] mca:base:select:(routed) Querying component [binomial]
[cn2:00817] [[15019,1],2] creating OOB-TCP module for interface eth0
[cn2:00817] [[15019,1],2] oob:tcp:init adding 192.168.1.2 to our list of V4 connections
[cn2:00817] [[15019,1],2] TCP STARTUP
[cn2:00817] [[15019,1],2] attempting to bind to IPv4 port 0
[cn2:00817] [[15019,1],2] assigned IPv4 port 55643
[cn2:00817] mca:oob:select: Adding component to end
[cn2:00817] mca:oob:select: Found 1 active transports
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 2335 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 880
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 880 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 263
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 263 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 304
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 304 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1523 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 880 bytes from [[15019,0],4] for tag 2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 263 bytes from [[15019,0],1] for tag 2
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 304 bytes from [[15019,0],3] for tag 2
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],2] for tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 2868 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 263
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 263 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 276
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 276 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14034] progress: initialized event flag to: 3
[cn1:14034] progress: initialized yield_when_idle to: false
[cn1:14034] progress: initialized num users to: 0
[cn1:14034] progress: initialized poll rate to: 23940000
[cn1:14034] progress: event_users_increment setting count to 1
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14017] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn5:14017] mca:base:select:(routed) Querying component [debruijn]
[cn5:14017] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn5:14017] mca:base:select:(routed) Querying component [direct]
[cn5:14017] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn5:14017] mca:base:select:(routed) Querying component [radix]
[cn5:14017] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn5:14017] mca:base:select:(routed) Selected component [binomial]
[cn5:14017] mca: base: close: unloading component debruijn
[cn5:14017] mca: base: close: unloading component direct
[cn5:14017] mca: base: close: unloading component radix
[cn5:14017] orte_routed_base_select: initializing selected component binomial
[cn2:00818] mca: base: components_register: registering rml components
[cn2:00818] mca: base: components_register: found loaded component oob
[cn2:00818] mca: base: components_register: component oob has no register or open function
[cn4:17680] mca: base: components_open: opening rml components
[cn4:17680] mca: base: components_open: found loaded component oob
[cn4:17680] mca: base: components_open: component oob open function successful
[cn4:17680] orte_rml_base_select: initializing rml component oob
[cn3:21268] mca: base: components_register: found loaded component bad
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1972
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1881 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 1972 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 877 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 304
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 304 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] message received 263 bytes from [[15019,0],3] for tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 276 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1880 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 1972 bytes from [[15019,0],4] for tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 304 bytes from [[15019,0],1] for tag 2
[cn4:17681] mca: base: components_register: registering rml components
[cn4:17681] mca: base: components_register: found loaded component oob
[cn4:17681] mca: base: components_register: component oob has no register or open function
[cn3:21269] mca: base: components_register: registering routed components
[cn3:21269] mca: base: components_register: found loaded component binomial
[cn3:21269] mca: base: components_register: component binomial has no register or open function
[cn5:14018] mca: base: components_register: registering routed components
[cn5:14018] mca: base: components_register: found loaded component binomial
[cn5:14018] mca: base: components_register: component binomial has no register or open function
[cn5:14018] mca: base: components_register: found loaded component debruijn
[cn5:14018] mca: base: components_register: component debruijn has no register or open function
[cn5:14018] mca: base: components_register: found loaded component direct
[cn5:14018] mca: base: components_register: component direct has no register or open function
[cn5:14018] mca: base: components_register: found loaded component radix
[cn5:14018] mca: base: components_register: component radix register function successful
[cn5:14018] mca: base: components_open: opening routed components
[cn5:14018] mca: base: components_open: found loaded component binomial
[cn5:14018] mca: base: components_open: found loaded component debruijn
[cn5:14018] mca: base: components_open: found loaded component direct
[cn5:14018] mca: base: components_open: found loaded component radix
[cn5:14018] mca:base:select: Auto-selecting routed components
[cn5:14018] mca:base:select:(routed) Querying component [binomial]
[cn5:14018] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn5:14018] mca:base:select:(routed) Querying component [debruijn]
[cn5:14018] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn5:14018] mca:base:select:(routed) Querying component [direct]
[cn5:14018] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn5:14018] mca:base:select:(routed) Querying component [radix]
[cn5:14018] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn5:14018] mca:base:select:(routed) Selected component [binomial]
[cn5:14018] mca: base: close: unloading component debruijn
[cn5:14018] mca: base: close: unloading component direct
[cn2:00818] mca: base: components_open: opening rml components
[cn2:00818] mca: base: components_open: found loaded component oob
[cn2:00818] mca: base: components_open: component oob open function successful
[cn2:00818] orte_rml_base_select: initializing rml component oob
[cn1:14033] [[15019,1],0]:tcp:recv:handler called for peer [[15019,0],0]
[cn1:14033] [[15019,1],0] RECV CONNECT ACK FROM [[15019,0],0] ON SOCKET 11
[cn1:14033] [[15019,1],0] waiting for connect ack from [[15019,0],0]
[cn1:14033] [[15019,1],0] connect ack received from [[15019,0],0]
[cn1:14033] [[15019,1],0] connect-ack recvd from [[15019,0],0]
[cn1:14033] [[15019,1],0] connect-ack header from [[15019,0],0] is okay
[cn1:14033] [[15019,1],0] waiting for connect ack from [[15019,0],0]
[cn1:14033] [[15019,1],0] connect ack received from [[15019,0],0]
[cn1:14033] [[15019,1],0] connect-ack version from [[15019,0],0] matches ours
[cn1:14033] [[15019,1],0] connect-ack [[15019,0],0] authenticated
[cn1:14033] [[15019,1],0]-[[15019,0],0] tcp_peer_connected on socket 11
[cn1:14033] [[15019,1],0]-[[15019,0],0] connected: 192.168.1.1 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:14033] [[15019,1],0]:tcp:recv:handler starting send/recv events
[cn1:14033] [[15019,1],0] tcp:set_module called for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 520
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 520 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1880 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 263
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 263 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1880 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 304
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 3035 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 304 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 443
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 443 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] message received 520 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 263 bytes from [[15019,0],1] for tag 2
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 304 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 443 bytes from [[15019,0],2] for tag 2
[cn5:14017] mca: base: components_register: registering dstore components
[cn5:14017] mca: base: components_register: found loaded component hash
[cn5:14017] mca: base: components_register: component hash register function successful
[cn5:14017] mca: base: components_open: opening dstore components
[cn5:14017] mca: base: components_open: found loaded component hash
[cn5:14017] mca:dstore:select: checking available component hash
[cn5:14017] mca:dstore:select: Querying component [hash]
[cn2:00817] mca: base: components_register: registering rml components
[cn2:00817] mca: base: components_register: found loaded component oob
[cn2:00817] mca: base: components_register: component oob has no register or open function
[cn4:17681] mca: base: components_open: opening rml components
[cn4:17681] mca: base: components_open: found loaded component oob
[cn4:17681] mca: base: components_open: component oob open function successful
[cn4:17681] orte_rml_base_select: initializing rml component oob
[cn3:21269] mca: base: components_register: found loaded component debruijn
[cn3:21269] mca: base: components_register: component debruijn has no register or open function
[cn3:21269] mca: base: components_register: found loaded component direct
[cn3:21269] mca: base: components_register: component direct has no register or open function
[cn3:21269] mca: base: components_register: found loaded component radix
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 164
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 164 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 304
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 304 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 104
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 104 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1369
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1369 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 164 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 304 bytes from [[15019,0],1] for tag 2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 104 bytes from [[15019,0],3] for tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 1369 bytes from [[15019,0],2] for tag 2
[cn5:14018] mca: base: close: unloading component radix
[cn5:14018] orte_routed_base_select: initializing selected component binomial
[cn2:00817] mca: base: components_open: opening rml components
[cn2:00817] mca: base: components_open: found loaded component oob
[cn2:00817] mca: base: components_open: component oob open function successful
[cn2:00817] orte_rml_base_select: initializing rml component oob
[cn4:17680] mca: base: components_register: registering routed components
[cn3:21269] mca: base: components_register: component radix register function successful
[cn3:21269] mca: base: components_open: opening routed components
[cn3:21269] mca: base: components_open: found loaded component binomial
[cn3:21269] mca: base: components_open: found loaded component debruijn
[cn3:21269] mca: base: components_open: found loaded component direct
[cn3:21269] mca: base: components_open: found loaded component radix
[cn3:21269] mca:base:select: Auto-selecting routed components
[cn3:21269] mca:base:select:(routed) Querying component [binomial]
[cn3:21269] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn3:21269] mca:base:select:(routed) Querying component [debruijn]
[cn3:21269] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn3:21269] mca:base:select:(routed) Querying component [direct]
[cn3:21269] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn3:21269] mca:base:select:(routed) Querying component [radix]
[cn3:21269] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn3:21269] mca:base:select:(routed) Selected component [binomial]
[cn3:21269] mca: base: close: unloading component debruijn
[cn3:21269] mca: base: close: unloading component direct
[cn3:21269] mca: base: close: unloading component radix
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 176
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 176 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 104
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 104 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 464 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 202
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 464 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1803 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 202 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 108
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 108 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 176 bytes from [[15019,0],4] for tag 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 464 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] message received 104 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 202 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 108 bytes from [[15019,0],2] for tag 2
[cn5:14018] mca: base: components_register: registering dstore components
[cn5:14018] mca: base: components_register: found loaded component hash
[cn2:00818] mca: base: components_register: registering routed components
[cn4:17680] mca: base: components_register: found loaded component binomial
[cn4:17680] mca: base: components_register: component binomial has no register or open function
[cn3:21269] orte_routed_base_select: initializing selected component binomial
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 374
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 374 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 616
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 616 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 443
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 443 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 121
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 121 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 374 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 616 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 443 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 121 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 192
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 192 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1256
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1256 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 738
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 738 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14018] mca: base: components_register: component hash register function successful
[cn5:14018] mca: base: components_open: opening dstore components
[cn5:14018] mca: base: components_open: found loaded component hash
[cn5:14018] mca:dstore:select: checking available component hash
[cn5:14018] mca:dstore:select: Querying component [hash]
[cn2:00817] mca: base: components_register: registering routed components
[cn2:00817] mca: base: components_register: found loaded component binomial
[cn2:00817] mca: base: components_register: component binomial has no register or open function
[cn2:00817] mca: base: components_register: found loaded component debruijn
[cn2:00817] mca: base: components_register: component debruijn has no register or open function
[cn2:00817] mca: base: components_register: found loaded component direct
[cn2:00817] mca: base: components_register: component direct has no register or open function
[cn4:17680] mca: base: components_register: found loaded component debruijn
[cn4:17680] mca: base: components_register: component debruijn has no register or open function
[cn4:17680] mca: base: components_register: found loaded component direct
[cn4:17680] mca: base: components_register: component direct has no register or open function
[cn4:17680] mca: base: components_register: found loaded component radix
[cn3:21268] mca: base: components_register: component bad has no register or open function
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1135
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 1135 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 192 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 1256 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 738 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] message received 1135 bytes from [[15019,0],4] for tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00817] mca: base: components_register: found loaded component radix
[cn2:00817] mca: base: components_register: component radix register function successful
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn4:17680] mca: base: components_register: component radix register function successful
[cn4:17680] mca: base: components_open: opening routed components
[cn4:17680] mca: base: components_open: found loaded component binomial
[cn4:17680] mca: base: components_open: found loaded component debruijn
[cn4:17680] mca: base: components_open: found loaded component direct
[cn4:17680] mca: base: components_open: found loaded component radix
[cn4:17680] mca:base:select: Auto-selecting routed components
[cn4:17680] mca:base:select:(routed) Querying component [binomial]
[cn4:17680] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn4:17680] mca:base:select:(routed) Querying component [debruijn]
[cn4:17680] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn4:17680] mca:base:select:(routed) Querying component [direct]
[cn4:17680] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn4:17680] mca:base:select:(routed) Querying component [radix]
[cn4:17680] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn4:17680] mca:base:select:(routed) Selected component [binomial]
[cn4:17680] mca: base: close: unloading component debruijn
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21268] mca: base: components_open: opening grpcomm components
[cn3:21268] mca: base: components_open: found loaded component bad
[cn3:21268] mca: base: components_open: component bad open function successful
[cn3:21268] mca:base:select: Auto-selecting grpcomm components
[cn3:21268] mca:base:select:(grpcomm) Querying component [bad]
[cn3:21268] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn3:21268] mca:base:select:(grpcomm) Selected component [bad]
[cn3:21268] [[15019,1],4] grpcomm:base:receive start comm
[cn3:21268] [[15019,1],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn3:21268] [[15019,1],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn5:14017] mca: base: components_register: registering grpcomm components
[cn5:14017] mca: base: components_register: found loaded component bad
[cn5:14017] mca: base: components_register: component bad has no register or open function
[cn5:14017] mca: base: components_open: opening grpcomm components
[cn5:14017] mca: base: components_open: found loaded component bad
[cn5:14017] mca: base: components_open: component bad open function successful
[cn5:14017] mca:base:select: Auto-selecting grpcomm components
[cn5:14017] mca:base:select:(grpcomm) Querying component [bad]
[cn5:14017] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn5:14017] mca:base:select:(grpcomm) Selected component [bad]
[cn5:14017] [[15019,1],8] grpcomm:base:receive start comm
[cn5:14017] [[15019,1],8] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn5:14017] [[15019,1],8] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn5:14017] [[15019,1],8] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn5:14017] [[15019,1],8] setting up session dir with
	tmpdir: /tmp
	host cn5
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1135
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 1135 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 100 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1280
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 1280 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 464 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 221
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 221 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 190
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 190 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 1135 bytes from [[15019,0],4] for tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 1280 bytes from [[15019,0],1] for tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 221 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] message received 190 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 69
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 2439 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 69 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 179 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 464 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 2016 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 108
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 108 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 276
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 276 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 520
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 520 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14018] mca: base: components_register: registering grpcomm components
[cn5:14018] mca: base: components_register: found loaded component bad
[cn5:14018] mca: base: components_register: component bad has no register or open function
[cn5:14018] mca: base: components_open: opening grpcomm components
[cn5:14018] mca: base: components_open: found loaded component bad
[cn5:14018] mca: base: components_open: component bad open function successful
[cn5:14018] mca:base:select: Auto-selecting grpcomm components
[cn5:14018] mca:base:select:(grpcomm) Querying component [bad]
[cn5:14018] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn5:14018] mca:base:select:(grpcomm) Selected component [bad]
[cn5:14018] [[15019,1],9] grpcomm:base:receive start comm
[cn5:14018] [[15019,1],9] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn5:14018] [[15019,1],9] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn5:14018] [[15019,1],9] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn5:14018] [[15019,1],9] setting up session dir with
	tmpdir: /tmp
	host cn5
[cn2:00817] mca: base: components_open: opening routed components
[cn2:00817] mca: base: components_open: found loaded component binomial
[cn2:00817] mca: base: components_open: found loaded component debruijn
[cn2:00817] mca: base: components_open: found loaded component direct
[cn2:00817] mca: base: components_open: found loaded component radix
[cn2:00817] mca:base:select: Auto-selecting routed components
[cn2:00817] mca:base:select:(routed) Querying component [binomial]
[cn2:00817] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn2:00817] mca:base:select:(routed) Querying component [debruijn]
[cn2:00817] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn2:00817] mca:base:select:(routed) Querying component [direct]
[cn2:00817] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn2:00817] mca:base:select:(routed) Querying component [radix]
[cn2:00817] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn2:00817] mca:base:select:(routed) Selected component [binomial]
[cn2:00817] mca: base: close: unloading component debruijn
[cn2:00817] mca: base: close: unloading component direct
[cn2:00817] mca: base: close: unloading component radix
[cn4:17680] mca: base: close: unloading component direct
[cn4:17680] mca: base: close: unloading component radix
[cn4:17680] orte_routed_base_select: initializing selected component binomial
[cn3:21268] [[15019,1],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn3:21268] [[15019,1],4] setting up session dir with
	tmpdir: /tmp
	host cn3
[cn1:14018] [[15019,0],0] message received 69 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 108 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 276 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 520 bytes from [[15019,0],2] for tag 2
[cn5:14018] [[15019,1],9] posting recv
[cn2:00817] orte_routed_base_select: initializing selected component binomial
[cn4:17681] mca: base: components_register: registering routed components
[cn4:17681] mca: base: components_register: found loaded component binomial
[cn4:17681] mca: base: components_register: component binomial has no register or open function
[cn3:21269] mca: base: components_register: registering dstore components
[cn3:21269] mca: base: components_register: found loaded component hash
[cn3:21269] mca: base: components_register: component hash register function successful
[cn3:21269] mca: base: components_open: opening dstore components
[cn3:21269] mca: base: components_open: found loaded component hash
[cn3:21269] mca:dstore:select: checking available component hash
[cn3:21269] mca:dstore:select: Querying component [hash]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 96
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 96 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 106
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 106 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 370
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 370 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1212
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1212 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 96 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 106 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 370 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 1212 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1056
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1056 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] procdir: /tmp/openmpi-sessions-artpol@cn5_0/15019/1/8
[cn2:00818] mca: base: components_register: found loaded component binomial
[cn4:17681] mca: base: components_register: found loaded component debruijn
[cn4:17681] mca: base: components_register: component debruijn has no register or open function
[cn4:17681] mca: base: components_register: found loaded component direct
[cn4:17681] mca: base: components_register: component direct has no register or open function
[cn3:21268] [[15019,1],4] posting recv
[cn3:21268] [[15019,1],4] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn3:21268] [[15019,1],4] posting recv
[cn3:21268] [[15019,1],4] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn3:21268] [[15019,1],4] posting recv
[cn3:21268] [[15019,1],4] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn3:21268] procdir: /tmp/openmpi-sessions-artpol@cn3_0/15019/1/4
[cn3:21268] jobdir: /tmp/openmpi-sessions-artpol@cn3_0/15019/1
[cn3:21268] top: openmpi-sessions-artpol@cn3_0
[cn3:21268] tmp: /tmp
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 18164154369
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 18164154369
[cn3:21268] [[15019,1],4] routed_binomial: init routes for proc job [15019,1]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
	daemon uri 984285184.2;tcp://192.168.1.3:38382
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 821
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 821 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1329
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1329 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 1056 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0] message received 821 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 126 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 1329 bytes from [[15019,0],3] for tag 2
[cn3:21268] [[15019,1],4] [[15019,0],2] with daemon registering
[cn3:21268] [[15019,1],4] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn3:21268] [[15019,1],4] rml_send_buffer to peer [[15019,0],2] at tag 1
[cn3:21268] [[15019,1],4] registering sync waiting for ack
[cn3:21268] [[15019,1],4] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn3:21268] [[15019,1],4]: set_addr to uri 984285184.2;tcp://192.168.1.3:38382
[cn3:21268] [[15019,1],4]:set_addr checking if peer [[15019,0],2] is reachable via component tcp
[cn3:21268] [[15019,1],4] oob:tcp: working peer [[15019,0],2] address tcp://192.168.1.3:38382
[cn3:21268] [[15019,1],4] PEER [[15019,0],2] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn3:21268] [[15019,1],4] PASSING ADDR 192.168.1.3 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21268] [[15019,1],4]:tcp set addr for peer [[15019,0],2]
[cn3:21268] [[15019,1],4]: peer [[15019,0],2] is reachable via component tcp
[cn5:14017] jobdir: /tmp/openmpi-sessions-artpol@cn5_0/15019/1
[cn5:14017] top: openmpi-sessions-artpol@cn5_0
[cn5:14017] tmp: /tmp
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 35344023553
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 35344023553
[cn5:14017] [[15019,1],8] routed_binomial: init routes for proc job [15019,1]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
	daemon uri 984285184.4;tcp://192.168.1.5:41719
[cn5:14017] [[15019,1],8] [[15019,0],4] with daemon registering
[cn2:00818] mca: base: components_register: component binomial has no register or open function
[cn4:17681] mca: base: components_register: found loaded component radix
[cn4:17681] mca: base: components_register: component radix register function successful
[cn4:17681] mca: base: components_open: opening routed components
[cn4:17681] mca: base: components_open: found loaded component binomial
[cn4:17681] mca: base: components_open: found loaded component debruijn
[cn4:17681] mca: base: components_open: found loaded component direct
[cn4:17681] mca: base: components_open: found loaded component radix
[cn4:17681] mca:base:select: Auto-selecting routed components
[cn4:17681] mca:base:select:(routed) Querying component [binomial]
[cn4:17681] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn4:17681] mca:base:select:(routed) Querying component [debruijn]
[cn4:17681] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn4:17681] mca:base:select:(routed) Querying component [direct]
[cn4:17681] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn4:17681] mca:base:select:(routed) Querying component [radix]
[cn4:17681] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn4:17681] mca:base:select:(routed) Selected component [binomial]
[cn4:17681] mca: base: close: unloading component debruijn
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 783
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 783 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 443
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 443 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 221
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 221 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 793
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 793 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 783 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 443 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 221 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 793 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1256
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 1256 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 104
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 104 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1193
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1193 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14017] [[15019,1],8] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn5:14017] [[15019,1],8] rml_send_buffer to peer [[15019,0],4] at tag 1
[cn5:14017] [[15019,1],8] registering sync waiting for ack
[cn5:14017] [[15019,1],8] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn5:14017] [[15019,1],8] posting recv
[cn5:14017] [[15019,1],8] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn5:14017] [[15019,1],8] posting recv
[cn5:14017] [[15019,1],8] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn5:14017] [[15019,1],8] posting recv
[cn5:14017] [[15019,1],8] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn2:00818] mca: base: components_register: found loaded component debruijn
[cn2:00818] mca: base: components_register: component debruijn has no register or open function
[cn2:00818] mca: base: components_register: found loaded component direct
[cn2:00818] mca: base: components_register: component direct has no register or open function
[cn2:00818] mca: base: components_register: found loaded component radix
[cn4:17681] mca: base: close: unloading component direct
[cn4:17681] mca: base: close: unloading component radix
[cn4:17681] orte_routed_base_select: initializing selected component binomial
[cn3:21268] [[15019,1],4] posting recv
[cn3:21268] [[15019,1],4] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn3:21268] [[15019,1],4] rml_send_msg to peer [[15019,0],2] at tag 1
[cn3:21268] [[15019,1],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21268] [[15019,1],4]:tcp:processing set_peer cmd for interface eth0
[cn3:21268] [[15019,1],4] SET_PEER ADDING PEER [[15019,0],2]
[cn3:21268] [[15019,1],4] set_peer: peer [[15019,0],2] is listening on interface eth0 net 192.168.1.3 port 38382
[cn3:21268] [[15019,1],4] oob:base:send to target [[15019,0],2]
[cn3:21268] [[15019,1],4] oob:tcp:send_nb to peer [[15019,0],2]:1
[cn3:21268] [[15019,1],4] tcp:send_nb to peer [[15019,0],2]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 551
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 551 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 2016 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 1256 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 104 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 1193 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 551 bytes from [[15019,0],4] for tag 2
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 221
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 221 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 102
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 102 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 743
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 743 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 174 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00818] mca: base: components_register: component radix register function successful
[cn2:00818] mca: base: components_open: opening routed components
[cn2:00818] mca: base: components_open: found loaded component binomial
[cn2:00818] mca: base: components_open: found loaded component debruijn
[cn2:00818] mca: base: components_open: found loaded component direct
[cn2:00818] mca: base: components_open: found loaded component radix
[cn2:00818] mca:base:select: Auto-selecting routed components
[cn2:00818] mca:base:select:(routed) Querying component [binomial]
[cn2:00818] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn2:00818] mca:base:select:(routed) Querying component [debruijn]
[cn2:00818] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn2:00818] mca:base:select:(routed) Querying component [direct]
[cn2:00818] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn2:00818] mca:base:select:(routed) Querying component [radix]
[cn2:00818] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn2:00818] mca:base:select:(routed) Selected component [binomial]
[cn2:00818] mca: base: close: unloading component debruijn
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17680] mca: base: components_register: registering dstore components
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn3:21268] [[15019,1],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],2]
[cn3:21268] [[15019,1],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],2]:1
[cn3:21268] [[15019,1],4] routed_binomial_get([[15019,0],2]) --> [[15019,0],2]
[cn3:21268] [[15019,1],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],2]
[cn3:21268] [[15019,1],4] tcp:send_nb: initiating connection to [[15019,0],2]
[cn3:21268] [[15019,1],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],2]
[cn3:21268] [[15019,1],4] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],2] via interface eth0
[cn3:21268] [[15019,1],4] oob:tcp:peer creating socket to [[15019,0],2]
[cn3:21268] [[15019,1],4] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],2] via interface eth0 on socket 11
[cn3:21268] [[15019,1],4] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],2] on 192.168.1.3:38382 - 0 retries
[cn5:14017] [[15019,1],8]: set_addr to uri 984285184.4;tcp://192.168.1.5:41719
[cn5:14017] [[15019,1],8]:set_addr checking if peer [[15019,0],4] is reachable via component tcp
[cn5:14017] [[15019,1],8] oob:tcp: working peer [[15019,0],4] address tcp://192.168.1.5:41719
[cn5:14017] [[15019,1],8] PEER [[15019,0],4] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:14017] [[15019,1],8] PASSING ADDR 192.168.1.5 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn5:14017] [[15019,1],8]:tcp set addr for peer [[15019,0],4]
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1204 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1374
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 1374 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 221 bytes from [[15019,0],1] for tag 2
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 102 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 743 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 1374 bytes from [[15019,0],4] for tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 787
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 855 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 787 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 104
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 104 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 855 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 2016 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 374
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 149 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 374 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 175 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 195
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 195 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00818] mca: base: close: unloading component direct
[cn2:00818] mca: base: close: unloading component radix
[cn2:00818] orte_routed_base_select: initializing selected component binomial
[cn4:17680] mca: base: components_register: found loaded component hash
[cn3:21268] [[15019,1],4] waiting for connect completion to [[15019,0],2] - activating send event
[cn3:21268] [[15019,1],4] tcp:send_handler called to send to peer [[15019,0],2]
[cn3:21268] [[15019,1],4] tcp:send_handler CONNECTING
[cn3:21268] [[15019,1],4]:tcp:complete_connect called for peer [[15019,0],2] on socket 11
[cn3:21268] [[15019,1],4] tcp_peer_complete_connect: sending ack to [[15019,0],2]
[cn3:21268] [[15019,1],4] SEND CONNECT ACK
[cn3:21268] fetching data from INTERNAL dstore
[cn3:21268] dstore:hash:fetch: searching for key opal.cred on proc 18164154369
[cn3:21268] dstore_hash:fetch key opal.cred for proc 18164154369 not found
[cn3:21268] [[15019,1],4] send blocking of 47 bytes to socket 11
[cn5:14017] [[15019,1],8]: peer [[15019,0],4] is reachable via component tcp
[cn5:14017] [[15019,1],8] posting recv
[cn5:14017] [[15019,1],8] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn5:14017] [[15019,1],8] rml_send_msg to peer [[15019,0],4] at tag 1
[cn5:14017] [[15019,1],8] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14017] [[15019,1],8]:tcp:processing set_peer cmd for interface eth0
[cn5:14017] [[15019,1],8] SET_PEER ADDING PEER [[15019,0],4]
[cn5:14017] [[15019,1],8] set_peer: peer [[15019,0],4] is listening on interface eth0 net 192.168.1.5 port 41719
[cn5:14017] [[15019,1],8] oob:base:send to target [[15019,0],4]
[cn5:14017] [[15019,1],8] oob:tcp:send_nb to peer [[15019,0],4]:1
[cn5:14017] [[15019,1],8] tcp:send_nb to peer [[15019,0],4]
[cn5:14017] [[15019,1],8]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],4]
[cn5:14017] [[15019,1],8]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],4]:1
[cn5:14017] [[15019,1],8] routed_binomial_get([[15019,0],4]) --> [[15019,0],4]
[cn5:14017] [[15019,1],8]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],4]
[cn1:14018] [[15019,0],0] message received 787 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 104 bytes from [[15019,0],1] for tag 2
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] message received 374 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] message received 195 bytes from [[15019,0],2] for tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 2016 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14017] [[15019,1],8] tcp:send_nb: initiating connection to [[15019,0],4]
[cn5:14017] [[15019,1],8]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],4]
[cn5:14017] [[15019,1],8] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],4] via interface eth0
[cn5:14017] [[15019,1],8] oob:tcp:peer creating socket to [[15019,0],4]
[cn5:14017] [[15019,1],8] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],4] via interface eth0 on socket 11
[cn5:14017] [[15019,1],8] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],4] on 192.168.1.5:41719 - 0 retries
[cn5:14017] [[15019,1],8] waiting for connect completion to [[15019,0],4] - activating send event
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00817] mca: base: components_register: registering dstore components
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17680] mca: base: components_register: component hash register function successful
[cn4:17680] mca: base: components_open: opening dstore components
[cn4:17680] mca: base: components_open: found loaded component hash
[cn4:17680] mca:dstore:select: checking available component hash
[cn4:17680] mca:dstore:select: Querying component [hash]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21268] [[15019,1],4] blocking send complete to socket 11
[cn3:21268] [[15019,1],4] tcp_peer_complete_connect: setting read event on connection to [[15019,0],2]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 645
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 645 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 102
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 102 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 104
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 104 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 998
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 998 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,1],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 74
[cn1:14018] [[15019,0],0] message received 645 bytes from [[15019,0],4] for tag 2
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 102 bytes from [[15019,0],1] for tag 2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 104 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 998 bytes from [[15019,0],2] for tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 387 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 104 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 195
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 195 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 374
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 374 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 102
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 102 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 494
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 494 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14033] [[15019,1],0] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14033] [[15019,1],0] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14033] [[15019,1],0] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 74 BYTES ON SOCKET 11
[cn1:14033] [[15019,1],0]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14034] mca: base: components_register: registering ess components
[cn1:14034] mca: base: components_register: found loaded component env
[cn1:14034] mca: base: components_register: component env has no register or open function
[cn1:14034] mca: base: components_open: opening ess components
[cn1:14034] mca: base: components_open: found loaded component env
[cn1:14034] mca: base: components_open: component env open function successful
[cn1:14034] mca:base:select: Auto-selecting ess components
[cn1:14034] mca:base:select:(  ess) Querying component [env]
[cn1:14034] mca:base:select:(  ess) Query of component [env] set priority to 20
[cn1:14034] mca:base:select:(  ess) Selected component [env]
[cn1:14034] ess:env set name to [[15019,1],1]
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14017] [[15019,1],8] tcp:send_handler called to send to peer [[15019,0],4]
[cn5:14017] [[15019,1],8] tcp:send_handler CONNECTING
[cn5:14017] [[15019,1],8]:tcp:complete_connect called for peer [[15019,0],4] on socket 11
[cn5:14017] [[15019,1],8] tcp_peer_complete_connect: sending ack to [[15019,0],4]
[cn5:14017] [[15019,1],8] SEND CONNECT ACK
[cn5:14017] fetching data from INTERNAL dstore
[cn5:14017] dstore:hash:fetch: searching for key opal.cred on proc 35344023553
[cn5:14017] dstore_hash:fetch key opal.cred for proc 35344023553 not found
[cn5:14017] [[15019,1],8] send blocking of 47 bytes to socket 11
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn2:00817] mca: base: components_register: found loaded component hash
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17681] mca: base: components_register: registering dstore components
[cn3:21268] [[15019,1],4]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21268] [[15019,1],4] RECV CONNECT ACK FROM [[15019,0],2] ON SOCKET 11
[cn3:21268] [[15019,1],4] waiting for connect ack from [[15019,0],2]
[cn3:21268] [[15019,1],4] connect ack received from [[15019,0],2]
[cn3:21268] [[15019,1],4] connect-ack recvd from [[15019,0],2]
[cn3:21268] [[15019,1],4] connect-ack header from [[15019,0],2] is okay
[cn3:21268] [[15019,1],4] waiting for connect ack from [[15019,0],2]
[cn3:21268] [[15019,1],4] connect ack received from [[15019,0],2]
[cn3:21268] [[15019,1],4] connect-ack version from [[15019,0],2] matches ours
[cn3:21268] [[15019,1],4] connect-ack [[15019,0],2] authenticated
[cn3:21268] [[15019,1],4]-[[15019,0],2] tcp_peer_connected on socket 11
[cn3:21268] [[15019,1],4]-[[15019,0],2] connected: 192.168.1.3 - 192.168.1.3 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn3:21268] [[15019,1],4]:tcp:recv:handler starting send/recv events
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 195 bytes from [[15019,0],4] for tag 2
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 120 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 374 bytes from [[15019,0],1] for tag 2
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 319 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] message received 102 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 494 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 104
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 104 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 374
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 374 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 267
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 267 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14017] [[15019,1],8] blocking send complete to socket 11
[cn5:14017] [[15019,1],8] tcp_peer_complete_connect: setting read event on connection to [[15019,0],4]
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00817] mca: base: components_register: component hash register function successful
[cn2:00817] mca: base: components_open: opening dstore components
[cn2:00817] mca: base: components_open: found loaded component hash
[cn2:00817] mca:dstore:select: checking available component hash
[cn2:00817] mca:dstore:select: Querying component [hash]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn4:17681] mca: base: components_register: found loaded component hash
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn3:21268] [[15019,1],4] tcp:set_module called for peer [[15019,0],2]
[cn3:21268] [[15019,1],4] tcp:send_handler called to send to peer [[15019,0],2]
[cn3:21268] [[15019,1],4] tcp:send_handler SENDING TO [[15019,0],2]
[cn3:21268] [[15019,1],4] MESSAGE SEND COMPLETE TO [[15019,0],2] OF 61 BYTES ON SOCKET 11
[cn3:21268] [[15019,1],4]-[[15019,0],2] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1462
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 319 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 464 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 1462 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 104 bytes from [[15019,0],1] for tag 2
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 102 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] message received 374 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 464 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 267 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 170 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 1462 bytes from [[15019,0],4] for tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 228
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 228 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 102
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 102 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 267
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 267 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 738
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 738 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00818] mca: base: components_register: registering dstore components
[cn4:17681] mca: base: components_register: component hash register function successful
[cn4:17681] mca: base: components_open: opening dstore components
[cn4:17681] mca: base: components_open: found loaded component hash
[cn4:17681] mca:dstore:select: checking available component hash
[cn4:17681] mca:dstore:select: Querying component [hash]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 120 BYTES ON SOCKET 9
[cn3:21269] mca: base: components_register: registering grpcomm components
[cn3:21269] mca: base: components_register: found loaded component bad
[cn3:21269] mca: base: components_register: component bad has no register or open function
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14017] [[15019,1],8]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14017] [[15019,1],8] RECV CONNECT ACK FROM [[15019,0],4] ON SOCKET 11
[cn5:14017] [[15019,1],8] waiting for connect ack from [[15019,0],4]
[cn5:14017] [[15019,1],8] connect ack received from [[15019,0],4]
[cn5:14017] [[15019,1],8] connect-ack recvd from [[15019,0],4]
[cn5:14017] [[15019,1],8] connect-ack header from [[15019,0],4] is okay
[cn5:14017] [[15019,1],8] waiting for connect ack from [[15019,0],4]
[cn5:14017] [[15019,1],8] connect ack received from [[15019,0],4]
[cn5:14017] [[15019,1],8] connect-ack version from [[15019,0],4] matches ours
[cn5:14017] [[15019,1],8] connect-ack [[15019,0],4] authenticated
[cn5:14017] [[15019,1],8]-[[15019,0],4] tcp_peer_connected on socket 11
[cn5:14017] [[15019,1],8]-[[15019,0],4] connected: 192.168.1.5 - 192.168.1.5 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn5:14017] [[15019,1],8]:tcp:recv:handler starting send/recv events
[cn5:14017] [[15019,1],8] tcp:set_module called for peer [[15019,0],4]
[cn5:14017] [[15019,1],8] tcp:send_handler called to send to peer [[15019,0],4]
[cn5:14017] [[15019,1],8] tcp:send_handler SENDING TO [[15019,0],4]
[cn5:14017] [[15019,1],8] MESSAGE SEND COMPLETE TO [[15019,0],4] OF 61 BYTES ON SOCKET 11
[cn5:14017] [[15019,1],8]-[[15019,0],4] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 228 bytes from [[15019,0],4] for tag 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 2016 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 102 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 267 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 2016 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 738 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 186 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 845
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 845 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 374
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 374 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 738
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 118 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 738 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 855 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 190
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 190 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14018] procdir: /tmp/openmpi-sessions-artpol@cn5_0/15019/1/9
[cn5:14018] jobdir: /tmp/openmpi-sessions-artpol@cn5_0/15019/1
[cn5:14018] top: openmpi-sessions-artpol@cn5_0
[cn5:14018] tmp: /tmp
[cn2:00818] mca: base: components_register: found loaded component hash
[cn4:17680] mca: base: components_register: registering grpcomm components
[cn4:17680] mca: base: components_register: found loaded component bad
[cn4:17680] mca: base: components_register: component bad has no register or open function
[cn3:21269] mca: base: components_open: opening grpcomm components
[cn3:21269] mca: base: components_open: found loaded component bad
[cn3:21269] mca: base: components_open: component bad open function successful
[cn3:21269] mca:base:select: Auto-selecting grpcomm components
[cn3:21269] mca:base:select:(grpcomm) Querying component [bad]
[cn3:21269] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn3:21269] mca:base:select:(grpcomm) Selected component [bad]
[cn3:21269] [[15019,1],5] grpcomm:base:receive start comm
[cn3:21269] [[15019,1],5] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn3:21269] [[15019,1],5] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn1:14018] [[15019,0],0] message received 845 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 374 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 738 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 190 bytes from [[15019,0],2] for tag 2
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 39638990849
[cn5:14018] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 39638990849
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 39638990849
[cn5:14018] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 39638990849
[cn5:14018] [[15019,1],9] routed_binomial: init routes for proc job [15019,1]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
	daemon uri 984285184.4;tcp://192.168.1.5:41719
[cn5:14018] [[15019,1],9] [[15019,0],4] with daemon registering
[cn5:14018] [[15019,1],9] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn5:14018] [[15019,1],9] rml_send_buffer to peer [[15019,0],4] at tag 1
[cn2:00818] mca: base: components_register: component hash register function successful
[cn2:00818] mca: base: components_open: opening dstore components
[cn2:00818] mca: base: components_open: found loaded component hash
[cn2:00818] mca:dstore:select: checking available component hash
[cn2:00818] mca:dstore:select: Querying component [hash]
[cn4:17680] mca: base: components_open: opening grpcomm components
[cn4:17680] mca: base: components_open: found loaded component bad
[cn4:17680] mca: base: components_open: component bad open function successful
[cn4:17680] mca:base:select: Auto-selecting grpcomm components
[cn4:17680] mca:base:select:(grpcomm) Querying component [bad]
[cn4:17680] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn4:17680] mca:base:select:(grpcomm) Selected component [bad]
[cn4:17680] [[15019,1],6] grpcomm:base:receive start comm
[cn4:17680] [[15019,1],6] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn4:17680] [[15019,1],6] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn3:21269] [[15019,1],5] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn3:21269] [[15019,1],5] setting up session dir with
	tmpdir: /tmp
	host cn3
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 858
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 855 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 858 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 187 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 105
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 105 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 190
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 190 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 100 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 160
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 160 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 858 bytes from [[15019,0],4] for tag 2
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 319 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 105 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 190 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 160 bytes from [[15019,0],2] for tag 2
[cn5:14018] [[15019,1],9] registering sync waiting for ack
[cn5:14018] [[15019,1],9] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn5:14018] [[15019,1],9] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn5:14018] [[15019,1],9] posting recv
[cn5:14018] [[15019,1],9] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn5:14018] [[15019,1],9] posting recv
[cn5:14018] [[15019,1],9] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn5:14018] [[15019,1],9]: set_addr to uri 984285184.4;tcp://192.168.1.5:41719
[cn5:14018] [[15019,1],9]:set_addr checking if peer [[15019,0],4] is reachable via component tcp
[cn5:14018] [[15019,1],9] oob:tcp: working peer [[15019,0],4] address tcp://192.168.1.5:41719
[cn2:00817] mca: base: components_register: registering grpcomm components
[cn4:17680] [[15019,1],6] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn4:17680] [[15019,1],6] setting up session dir with
	tmpdir: /tmp
	host cn4
[cn3:21269] [[15019,1],5] posting recv
[cn3:21269] [[15019,1],5] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 995
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 995 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 192
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 192 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 187 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 105
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 319 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 105 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 69
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 69 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 995 bytes from [[15019,0],4] for tag 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 116 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 192 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 105 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 69 bytes from [[15019,0],2] for tag 2
[cn5:14018] [[15019,1],9] PEER [[15019,0],4] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn5:14018] [[15019,1],9] PASSING ADDR 192.168.1.5 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn5:14018] [[15019,1],9]:tcp set addr for peer [[15019,0],4]
[cn5:14018] [[15019,1],9]: peer [[15019,0],4] is reachable via component tcp
[cn5:14018] [[15019,1],9] posting recv
[cn5:14018] [[15019,1],9] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn5:14018] [[15019,1],9] rml_send_msg to peer [[15019,0],4] at tag 1
[cn5:14018] [[15019,1],9] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14018] [[15019,1],9]:tcp:processing set_peer cmd for interface eth0
[cn5:14018] [[15019,1],9] SET_PEER ADDING PEER [[15019,0],4]
[cn5:14018] [[15019,1],9] set_peer: peer [[15019,0],4] is listening on interface eth0 net 192.168.1.5 port 41719
[cn5:14018] [[15019,1],9] oob:base:send to target [[15019,0],4]
[cn2:00817] mca: base: components_register: found loaded component bad
[cn2:00817] mca: base: components_register: component bad has no register or open function
[cn4:17681] mca: base: components_register: registering grpcomm components
[cn3:21269] [[15019,1],5] posting recv
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1319
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 1319 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 158 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 738
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 738 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 99 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 639
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 639 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] message received 1319 bytes from [[15019,0],4] for tag 2
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 738 bytes from [[15019,0],1] for tag 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 639 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 743
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 743 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 186 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 190
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 190 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 419
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 419 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 102 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1132
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1132 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 187 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14018] [[15019,1],9] oob:tcp:send_nb to peer [[15019,0],4]:1
[cn5:14018] [[15019,1],9] tcp:send_nb to peer [[15019,0],4]
[cn5:14018] [[15019,1],9]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],4]
[cn5:14018] [[15019,1],9]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],4]:1
[cn5:14018] [[15019,1],9] routed_binomial_get([[15019,0],4]) --> [[15019,0],4]
[cn5:14018] [[15019,1],9]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],4]
[cn5:14018] [[15019,1],9] tcp:send_nb: initiating connection to [[15019,0],4]
[cn5:14018] [[15019,1],9]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],4]
[cn5:14018] [[15019,1],9] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],4] via interface eth0
[cn5:14018] [[15019,1],9] oob:tcp:peer creating socket to [[15019,0],4]
[cn5:14018] [[15019,1],9] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],4] via interface eth0 on socket 11
[cn5:14018] [[15019,1],9] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],4] on 192.168.1.5:41719 - 0 retries
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00817] mca: base: components_open: opening grpcomm components
[cn2:00817] mca: base: components_open: found loaded component bad
[cn2:00817] mca: base: components_open: component bad open function successful
[cn2:00817] mca:base:select: Auto-selecting grpcomm components
[cn2:00817] mca:base:select:(grpcomm) Querying component [bad]
[cn2:00817] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn2:00817] mca:base:select:(grpcomm) Selected component [bad]
[cn2:00817] [[15019,1],2] grpcomm:base:receive start comm
[cn2:00817] [[15019,1],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn2:00817] [[15019,1],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn4:17681] mca: base: components_register: found loaded component bad
[cn3:21269] procdir: /tmp/openmpi-sessions-artpol@cn3_0/15019/1/5
[cn3:21269] jobdir: /tmp/openmpi-sessions-artpol@cn3_0/15019/1
[cn3:21269] top: openmpi-sessions-artpol@cn3_0
[cn3:21269] tmp: /tmp
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 22459121665
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 22459121665
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 103 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 743 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 190 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 419 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 1132 bytes from [[15019,0],2] for tag 2
[cn5:14018] [[15019,1],9] waiting for connect completion to [[15019,0],4] - activating send event
[cn5:14018] [[15019,1],9] tcp:send_handler called to send to peer [[15019,0],4]
[cn5:14018] [[15019,1],9] tcp:send_handler CONNECTING
[cn5:14018] [[15019,1],9]:tcp:complete_connect called for peer [[15019,0],4] on socket 11
[cn5:14018] [[15019,1],9] tcp_peer_complete_connect: sending ack to [[15019,0],4]
[cn5:14018] [[15019,1],9] SEND CONNECT ACK
[cn5:14018] fetching data from INTERNAL dstore
[cn5:14018] dstore:hash:fetch: searching for key opal.cred on proc 39638990849
[cn5:14018] dstore_hash:fetch key opal.cred for proc 39638990849 not found
[cn5:14018] [[15019,1],9] send blocking of 47 bytes to socket 11
[cn2:00817] [[15019,1],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn2:00817] [[15019,1],2] setting up session dir with
	tmpdir: /tmp
	host cn2
[cn4:17680] [[15019,1],6] posting recv
[cn4:17680] [[15019,1],6] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn4:17680] [[15019,1],6] posting recv
[cn4:17680] [[15019,1],6] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn4:17680] [[15019,1],6] posting recv
[cn4:17680] [[15019,1],6] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21269] [[15019,1],5] routed_binomial: init routes for proc job [15019,1]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
	daemon uri 984285184.2;tcp://192.168.1.3:38382
[cn3:21269] [[15019,1],5] [[15019,0],2] with daemon registering
[cn3:21269] [[15019,1],5] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn3:21269] [[15019,1],5] rml_send_buffer to peer [[15019,0],2] at tag 1
[cn3:21269] [[15019,1],5] registering sync waiting for ack
[cn3:21269] [[15019,1],5] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn3:21269] [[15019,1],5] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn3:21269] [[15019,1],5] posting recv
[cn3:21269] [[15019,1],5] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn3:21269] [[15019,1],5]: set_addr to uri 984285184.2;tcp://192.168.1.3:38382
[cn3:21269] [[15019,1],5]:set_addr checking if peer [[15019,0],2] is reachable via component tcp
[cn3:21269] [[15019,1],5] oob:tcp: working peer [[15019,0],2] address tcp://192.168.1.3:38382
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 195
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 195 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 105
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 105 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 119 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 187 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 121
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 121 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1406
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1406 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] message received 195 bytes from [[15019,0],4] for tag 2
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 105 bytes from [[15019,0],1] for tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 103 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 121 bytes from [[15019,0],3] for tag 2
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 1406 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14018] [[15019,1],9] blocking send complete to socket 11
[cn5:14018] [[15019,1],9] tcp_peer_complete_connect: setting read event on connection to [[15019,0],4]
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00818] mca: base: components_register: registering grpcomm components
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17681] mca: base: components_register: component bad has no register or open function
[cn3:21269] [[15019,1],5] PEER [[15019,0],2] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn3:21269] [[15019,1],5] PASSING ADDR 192.168.1.3 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn3:21269] [[15019,1],5]:tcp set addr for peer [[15019,0],2]
[cn3:21269] [[15019,1],5]: peer [[15019,0],2] is reachable via component tcp
[cn3:21269] [[15019,1],5] posting recv
[cn3:21269] [[15019,1],5] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn3:21269] [[15019,1],5] rml_send_msg to peer [[15019,0],2] at tag 1
[cn3:21269] [[15019,1],5] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21269] [[15019,1],5]:tcp:processing set_peer cmd for interface eth0
[cn3:21269] [[15019,1],5] SET_PEER ADDING PEER [[15019,0],2]
[cn3:21269] [[15019,1],5] set_peer: peer [[15019,0],2] is listening on interface eth0 net 192.168.1.3 port 38382
[cn3:21269] [[15019,1],5] oob:base:send to target [[15019,0],2]
[cn3:21269] [[15019,1],5] oob:tcp:send_nb to peer [[15019,0],2]:1
[cn3:21269] [[15019,1],5] tcp:send_nb to peer [[15019,0],2]
[cn3:21269] [[15019,1],5]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],2]
[cn3:21269] [[15019,1],5]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],2]:1
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1149
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 1149 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 158 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 693 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1251
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1251 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1481
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1481 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] message received 1149 bytes from [[15019,0],4] for tag 2
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 319 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 1251 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 1481 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 343
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 343 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 319 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 290
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 328 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 290 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 898
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 111 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 898 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 618
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 118 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 618 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14018] [[15019,1],9]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14018] [[15019,1],9] RECV CONNECT ACK FROM [[15019,0],4] ON SOCKET 11
[cn5:14018] [[15019,1],9] waiting for connect ack from [[15019,0],4]
[cn5:14018] [[15019,1],9] connect ack received from [[15019,0],4]
[cn5:14018] [[15019,1],9] connect-ack recvd from [[15019,0],4]
[cn5:14018] [[15019,1],9] connect-ack header from [[15019,0],4] is okay
[cn5:14018] [[15019,1],9] waiting for connect ack from [[15019,0],4]
[cn5:14018] [[15019,1],9] connect ack received from [[15019,0],4]
[cn5:14018] [[15019,1],9] connect-ack version from [[15019,0],4] matches ours
[cn5:14018] [[15019,1],9] connect-ack [[15019,0],4] authenticated
[cn5:14018] [[15019,1],9]-[[15019,0],4] tcp_peer_connected on socket 11
[cn5:14018] [[15019,1],9]-[[15019,0],4] connected: 192.168.1.5 - 192.168.1.5 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn5:14018] [[15019,1],9]:tcp:recv:handler starting send/recv events
[cn5:14018] [[15019,1],9] tcp:set_module called for peer [[15019,0],4]
[cn5:14018] [[15019,1],9] tcp:send_handler called to send to peer [[15019,0],4]
[cn2:00818] mca: base: components_register: found loaded component bad
[cn4:17680] procdir: /tmp/openmpi-sessions-artpol@cn4_0/15019/1/6
[cn4:17680] jobdir: /tmp/openmpi-sessions-artpol@cn4_0/15019/1
[cn4:17680] top: openmpi-sessions-artpol@cn4_0
[cn4:17680] tmp: /tmp
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 26754088961
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 26754088961
[cn4:17680] [[15019,1],6] routed_binomial: init routes for proc job [15019,1]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
	daemon uri 984285184.3;tcp://192.168.1.4:43236
[cn4:17680] [[15019,1],6] [[15019,0],3] with daemon registering
[cn4:17680] [[15019,1],6] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn4:17680] [[15019,1],6] rml_send_buffer to peer [[15019,0],3] at tag 1
[cn4:17680] [[15019,1],6] registering sync waiting for ack
[cn4:17680] [[15019,1],6] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn3:21269] [[15019,1],5] routed_binomial_get([[15019,0],2]) --> [[15019,0],2]
[cn3:21269] [[15019,1],5]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],2]
[cn3:21269] [[15019,1],5] tcp:send_nb: initiating connection to [[15019,0],2]
[cn3:21269] [[15019,1],5]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],2]
[cn3:21269] [[15019,1],5] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],2] via interface eth0
[cn3:21269] [[15019,1],5] oob:tcp:peer creating socket to [[15019,0],2]
[cn3:21269] [[15019,1],5] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],2] via interface eth0 on socket 11
[cn3:21269] [[15019,1],5] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],2] on 192.168.1.3:38382 - 0 retries
[cn3:21269] [[15019,1],5] waiting for connect completion to [[15019,0],2] - activating send event
[cn3:21269] [[15019,1],5] tcp:send_handler called to send to peer [[15019,0],2]
[cn3:21269] [[15019,1],5] tcp:send_handler CONNECTING
[cn3:21269] [[15019,1],5]:tcp:complete_connect called for peer [[15019,0],2] on socket 11
[cn3:21269] [[15019,1],5] tcp_peer_complete_connect: sending ack to [[15019,0],2]
[cn3:21269] [[15019,1],5] SEND CONNECT ACK
[cn3:21269] fetching data from INTERNAL dstore
[cn3:21269] dstore:hash:fetch: searching for key opal.cred on proc 22459121665
[cn1:14018] [[15019,0],0] message received 343 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 290 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 898 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 618 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 351
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 100 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 351 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 159
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 159 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 116 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 109
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 109 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1179
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1179 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14018] [[15019,1],9] tcp:send_handler SENDING TO [[15019,0],4]
[cn5:14018] [[15019,1],9] MESSAGE SEND COMPLETE TO [[15019,0],4] OF 61 BYTES ON SOCKET 11
[cn5:14018] [[15019,1],9]-[[15019,0],4] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00817] [[15019,1],2] posting recv
[cn2:00817] [[15019,1],2] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn2:00817] [[15019,1],2] posting recv
[cn2:00817] [[15019,1],2] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn4:17681] mca: base: components_open: opening grpcomm components
[cn4:17681] mca: base: components_open: found loaded component bad
[cn4:17681] mca: base: components_open: component bad open function successful
[cn4:17681] mca:base:select: Auto-selecting grpcomm components
[cn4:17681] mca:base:select:(grpcomm) Querying component [bad]
[cn4:17681] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn4:17681] mca:base:select:(grpcomm) Selected component [bad]
[cn4:17681] [[15019,1],7] grpcomm:base:receive start comm
[cn4:17681] [[15019,1],7] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn4:17681] [[15019,1],7] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn4:17681] [[15019,1],7] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn4:17681] [[15019,1],7] setting up session dir with
	tmpdir: /tmp
	host cn4
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21269] dstore_hash:fetch key opal.cred for proc 22459121665 not found
[cn3:21269] [[15019,1],5] send blocking of 47 bytes to socket 11
[cn3:21269] [[15019,1],5] blocking send complete to socket 11
[cn3:21269] [[15019,1],5] tcp_peer_complete_connect: setting read event on connection to [[15019,0],2]
[cn3:21269] [[15019,1],5]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21269] [[15019,1],5] RECV CONNECT ACK FROM [[15019,0],2] ON SOCKET 11
[cn3:21269] [[15019,1],5] waiting for connect ack from [[15019,0],2]
[cn3:21269] [[15019,1],5] connect ack received from [[15019,0],2]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 351 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 103 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 159 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 109 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 1179 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 179
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 179 BYTES FOR DEST [[15019,0],0] TAG 5
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 121
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 121 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 119 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 127
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 103 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 127 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 273
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 119 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 273 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14017] [[15019,1],8]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14017] [[15019,1],8]:tcp:recv:handler CONNECTED
[cn5:14017] [[15019,1],8]:tcp:recv:handler allocate new recv msg
[cn5:14017] [[15019,1],8]:tcp:recv:handler read hdr
[cn5:14017] [[15019,1],8]:tcp:recv:handler allocate data region of size 14140
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn2:00817] [[15019,1],2] posting recv
[cn2:00817] [[15019,1],2] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn4:17680] [[15019,1],6]: set_addr to uri 984285184.3;tcp://192.168.1.4:43236
[cn3:21269] [[15019,1],5] connect-ack recvd from [[15019,0],2]
[cn3:21269] [[15019,1],5] connect-ack header from [[15019,0],2] is okay
[cn3:21269] [[15019,1],5] waiting for connect ack from [[15019,0],2]
[cn3:21269] [[15019,1],5] connect ack received from [[15019,0],2]
[cn3:21269] [[15019,1],5] connect-ack version from [[15019,0],2] matches ours
[cn3:21269] [[15019,1],5] connect-ack [[15019,0],2] authenticated
[cn3:21269] [[15019,1],5]-[[15019,0],2] tcp_peer_connected on socket 11
[cn3:21269] [[15019,1],5]-[[15019,0],2] connected: 192.168.1.3 - 192.168.1.3 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn3:21269] [[15019,1],5]:tcp:recv:handler starting send/recv events
[cn3:21269] [[15019,1],5] tcp:set_module called for peer [[15019,0],2]
[cn3:21269] [[15019,1],5] tcp:send_handler called to send to peer [[15019,0],2]
[cn3:21269] [[15019,1],5] tcp:send_handler SENDING TO [[15019,0],2]
[cn3:21269] [[15019,1],5] MESSAGE SEND COMPLETE TO [[15019,0],2] OF 61 BYTES ON SOCKET 11
[cn3:21269] [[15019,1],5]-[[15019,0],2] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] message received 179 bytes from [[15019,0],4] for tag 5
[cn1:14018] [[15019,0],0] plm:base:receive processing msg
[cn1:14018] [[15019,0],0] routed_binomial: init routes for HNP job [15019,1]
[cn1:14018] [[15019,0],0] routed_binomial:callback got uri 984285185.8;tcp://192.168.1.5:52061 for job [15019,1] rank 8
[cn1:14018] [[15019,0],0] routed_binomial:callback got uri 984285185.9;tcp://192.168.1.5:42002 for job [15019,1] rank 9
[cn1:14018] [[15019,0],0] plm:base:receive done processing commands
[cn1:14018] [[15019,0],0] message received 121 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 127 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 273 bytes from [[15019,0],2] for tag 2
[cn2:00818] mca: base: components_register: component bad has no register or open function
[cn4:17680] [[15019,1],6]:set_addr checking if peer [[15019,0],3] is reachable via component tcp
[cn3:21268] [[15019,1],4]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21268] [[15019,1],4]:tcp:recv:handler CONNECTED
[cn3:21268] [[15019,1],4]:tcp:recv:handler allocate new recv msg
[cn3:21268] [[15019,1],4]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 103
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 103 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 335
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 335 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 419
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 419 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 108
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 108 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 103 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 335 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 419 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 108 bytes from [[15019,0],2] for tag 2
[cn5:14018] [[15019,1],9]:tcp:recv:handler called for peer [[15019,0],4]
[cn2:00817] procdir: /tmp/openmpi-sessions-artpol@cn2_0/15019/1/2
[cn2:00817] jobdir: /tmp/openmpi-sessions-artpol@cn2_0/15019/1
[cn2:00817] top: openmpi-sessions-artpol@cn2_0
[cn2:00817] tmp: /tmp
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn4:17681] [[15019,1],7] posting recv
[cn4:17681] [[15019,1],7] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn4:17681] [[15019,1],7] posting recv
[cn4:17681] [[15019,1],7] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn4:17681] [[15019,1],7] posting recv
[cn4:17681] [[15019,1],7] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn3:21268] [[15019,1],4]:tcp:recv:handler allocate data region of size 14140
[cn1:14034] mca: base: components_register: registering oob components
[cn1:14034] mca: base: components_register: found loaded component tcp
[cn1:14034] mca: base: components_register: component tcp register function successful
[cn1:14034] mca: base: components_open: opening oob components
[cn1:14034] mca: base: components_open: found loaded component tcp
[cn1:14034] mca: base: components_open: component tcp open function successful
[cn1:14034] mca:oob:select: checking available component tcp
[cn1:14034] mca:oob:select: Querying component [tcp]
[cn1:14034] oob:tcp: component_available called
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 83 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 578
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 578 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1251
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1251 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 179
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 179 BYTES FOR DEST [[15019,0],0] TAG 5
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 83 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 578 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 1251 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 179 bytes from [[15019,0],2] for tag 5
[cn1:14018] [[15019,0],0] plm:base:receive processing msg
[cn1:14018] [[15019,0],0] routed_binomial: init routes for HNP job [15019,1]
[cn1:14018] [[15019,0],0] routed_binomial:callback got uri 984285185.4;tcp://192.168.1.3:53220 for job [15019,1] rank 4
[cn1:14018] [[15019,0],0] routed_binomial:callback got uri 984285185.5;tcp://192.168.1.3:48732 for job [15019,1] rank 5
[cn1:14018] [[15019,0],0] plm:base:receive done processing commands
[cn5:14018] [[15019,1],9]:tcp:recv:handler CONNECTED
[cn2:00817] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 9574219777
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn2:00817] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 9574219777
[cn2:00817] [[15019,1],2] routed_binomial: init routes for proc job [15019,1]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
	daemon uri 984285184.1;tcp://192.168.1.2:49722
[cn2:00817] [[15019,1],2] [[15019,0],1] with daemon registering
[cn4:17681] procdir: /tmp/openmpi-sessions-artpol@cn4_0/15019/1/7
[cn4:17681] jobdir: /tmp/openmpi-sessions-artpol@cn4_0/15019/1
[cn4:17681] top: openmpi-sessions-artpol@cn4_0
[cn4:17681] tmp: /tmp
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 31049056257
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 31049056257
[cn4:17681] [[15019,1],7] routed_binomial: init routes for proc job [15019,1]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
	daemon uri 984285184.3;tcp://192.168.1.4:43236
[cn4:17681] [[15019,1],7] [[15019,0],3] with daemon registering
[cn4:17681] [[15019,1],7] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn4:17681] [[15019,1],7] rml_send_buffer to peer [[15019,0],3] at tag 1
[cn4:17681] [[15019,1],7] registering sync waiting for ack
[cn4:17681] [[15019,1],7] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 95
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 95 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 664
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 664 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 313 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 851
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 851 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 103
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 321 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 103 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 95 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 664 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 851 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 103 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 82
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 82 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 409 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 818
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 818 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1557
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1557 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 170 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 278
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 278 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14034] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4
[cn1:14034] [[15019,1],1] oob:tcp:init rejecting loopback interface lo
[cn1:14034] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4
[cn1:14034] [[15019,1],1] oob:tcp:init creating module for V4 address on interface eth0
[cn1:14034] [[15019,1],1] creating OOB-TCP module for interface eth0
[cn1:14034] [[15019,1],1] oob:tcp:init adding 192.168.1.1 to our list of V4 connections
[cn1:14034] WORKING INTERFACE 3 KERNEL INDEX 3 FAMILY: V4
[cn1:14034] [[15019,1],1] oob:tcp:init creating module for V4 address on interface eth1
[cn1:14034] [[15019,1],1] creating OOB-TCP module for interface eth1
[cn1:14034] [[15019,1],1] oob:tcp:init adding 192.168.10.2 to our list of V4 connections
[cn1:14034] [[15019,1],1] TCP STARTUP
[cn1:14034] [[15019,1],1] attempting to bind to IPv4 port 0
[cn1:14034] [[15019,1],1] assigned IPv4 port 50040
[cn1:14034] mca:oob:select: Adding component to end
[cn1:14034] mca:oob:select: Found 1 active transports
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 186 BYTES ON SOCKET 9
[cn5:14018] [[15019,1],9]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00817] [[15019,1],2] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn2:00817] [[15019,1],2] rml_send_buffer to peer [[15019,0],1] at tag 1
[cn2:00817] [[15019,1],2] registering sync waiting for ack
[cn2:00817] [[15019,1],2] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn2:00817] [[15019,1],2]: set_addr to uri 984285184.1;tcp://192.168.1.2:49722
[cn2:00817] [[15019,1],2]:set_addr checking if peer [[15019,0],1] is reachable via component tcp
[cn2:00817] [[15019,1],2] oob:tcp: working peer [[15019,0],1] address tcp://192.168.1.2:49722
[cn4:17680] [[15019,1],6] oob:tcp: working peer [[15019,0],3] address tcp://192.168.1.4:43236
[cn4:17680] [[15019,1],6] PEER [[15019,0],3] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17680] [[15019,1],6] PASSING ADDR 192.168.1.4 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn4:17680] [[15019,1],6]:tcp set addr for peer [[15019,0],3]
[cn4:17680] [[15019,1],6]: peer [[15019,0],3] is reachable via component tcp
[cn4:17680] [[15019,1],6] posting recv
[cn4:17680] [[15019,1],6] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn4:17680] [[15019,1],6] rml_send_msg to peer [[15019,0],3] at tag 1
[cn4:17680] [[15019,1],6] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17680] [[15019,1],6]:tcp:processing set_peer cmd for interface eth0
[cn3:21269] [[15019,1],5]:tcp:recv:handler called for peer [[15019,0],2]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 82 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 818 bytes from [[15019,0],1] for tag 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] message received 1557 bytes from [[15019,0],3] for tag 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 278 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 273 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14018] [[15019,1],9]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00817] [[15019,1],2] PEER [[15019,0],1] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn2:00817] [[15019,1],2] PASSING ADDR 192.168.1.2 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00817] [[15019,1],2]:tcp set addr for peer [[15019,0],1]
[cn2:00817] [[15019,1],2]: peer [[15019,0],1] is reachable via component tcp
[cn2:00817] [[15019,1],2] posting recv
[cn2:00817] [[15019,1],2] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn2:00817] [[15019,1],2] rml_send_msg to peer [[15019,0],1] at tag 1
[cn2:00817] [[15019,1],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00817] [[15019,1],2]:tcp:processing set_peer cmd for interface eth0
[cn2:00817] [[15019,1],2] SET_PEER ADDING PEER [[15019,0],1]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17680] [[15019,1],6] SET_PEER ADDING PEER [[15019,0],3]
[cn4:17680] [[15019,1],6] set_peer: peer [[15019,0],3] is listening on interface eth0 net 192.168.1.4 port 43236
[cn4:17680] [[15019,1],6] oob:base:send to target [[15019,0],3]
[cn4:17680] [[15019,1],6] oob:tcp:send_nb to peer [[15019,0],3]:1
[cn4:17680] [[15019,1],6] tcp:send_nb to peer [[15019,0],3]
[cn4:17680] [[15019,1],6]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],3]
[cn4:17680] [[15019,1],6]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],3]:1
[cn4:17680] [[15019,1],6] routed_binomial_get([[15019,0],3]) --> [[15019,0],3]
[cn4:17680] [[15019,1],6]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],3]
[cn4:17680] [[15019,1],6] tcp:send_nb: initiating connection to [[15019,0],3]
[cn4:17680] [[15019,1],6]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],3]
[cn4:17680] [[15019,1],6] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],3] via interface eth0
[cn4:17680] [[15019,1],6] oob:tcp:peer creating socket to [[15019,0],3]
[cn4:17680] [[15019,1],6] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],3] via interface eth0 on socket 11
[cn4:17680] [[15019,1],6] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],3] on 192.168.1.4:43236 - 0 retries
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21269] [[15019,1],5]:tcp:recv:handler CONNECTED
[cn3:21269] [[15019,1],5]:tcp:recv:handler allocate new recv msg
[cn3:21269] [[15019,1],5]:tcp:recv:handler read hdr
[cn3:21269] [[15019,1],5]:tcp:recv:handler allocate data region of size 14140
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 108
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 108 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1165
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 1165 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 245 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 908
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 908 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 156
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 156 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 108 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 1165 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 908 bytes from [[15019,0],3] for tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 156 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14034] mca: base: components_register: registering rml components
[cn1:14034] mca: base: components_register: found loaded component oob
[cn1:14034] mca: base: components_register: component oob has no register or open function
[cn1:14034] mca: base: components_open: opening rml components
[cn1:14034] mca: base: components_open: found loaded component oob
[cn1:14034] mca: base: components_open: component oob open function successful
[cn1:14034] orte_rml_base_select: initializing rml component oob
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14018] [[15019,1],9]:tcp:recv:handler allocate data region of size 14140
[cn2:00817] [[15019,1],2] set_peer: peer [[15019,0],1] is listening on interface eth0 net 192.168.1.2 port 49722
[cn2:00817] [[15019,1],2] oob:base:send to target [[15019,0],1]
[cn2:00817] [[15019,1],2] oob:tcp:send_nb to peer [[15019,0],1]:1
[cn2:00817] [[15019,1],2] tcp:send_nb to peer [[15019,0],1]
[cn2:00817] [[15019,1],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],1]
[cn2:00817] [[15019,1],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],1]:1
[cn2:00817] [[15019,1],2] routed_binomial_get([[15019,0],1]) --> [[15019,0],1]
[cn2:00817] [[15019,1],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],1]
[cn2:00817] [[15019,1],2] tcp:send_nb: initiating connection to [[15019,0],1]
[cn2:00817] [[15019,1],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],1]
[cn2:00817] [[15019,1],2] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],1] via interface eth0
[cn4:17680] [[15019,1],6] waiting for connect completion to [[15019,0],3] - activating send event
[cn4:17680] [[15019,1],6] tcp:send_handler called to send to peer [[15019,0],3]
[cn4:17680] [[15019,1],6] tcp:send_handler CONNECTING
[cn4:17680] [[15019,1],6]:tcp:complete_connect called for peer [[15019,0],3] on socket 11
[cn4:17680] [[15019,1],6] tcp_peer_complete_connect: sending ack to [[15019,0],3]
[cn4:17680] [[15019,1],6] SEND CONNECT ACK
[cn4:17680] fetching data from INTERNAL dstore
[cn4:17680] dstore:hash:fetch: searching for key opal.cred on proc 26754088961
[cn4:17680] dstore_hash:fetch key opal.cred for proc 26754088961 not found
[cn4:17680] [[15019,1],6] send blocking of 47 bytes to socket 11
[cn4:17680] [[15019,1],6] blocking send complete to socket 11
[cn4:17680] [[15019,1],6] tcp_peer_complete_connect: setting read event on connection to [[15019,0],3]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21268] [[15019,1],4]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21268] [[15019,1],4]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 170 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 608
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 608 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 683
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 186 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 683 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1149
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1149 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 482
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 482 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 608 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 683 bytes from [[15019,0],1] for tag 2
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 187 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 1149 bytes from [[15019,0],3] for tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 482 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14017] [[15019,1],8]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14017] [[15019,1],8]:tcp:recv:handler CONNECTED
[cn5:14017] [[15019,1],8] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 14140 BYTES FOR DEST [[15019,1],8] TAG 20
[cn5:14017] [[15019,1],8] DELIVERING TO RML
[cn5:14017] [[15019,1],8] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] [[15019,1],8] message received 14140 bytes from [[15019,0],4] for tag 20
[cn5:14017] [[15019,1],8] registering sync ack recvd
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00817] [[15019,1],2] oob:tcp:peer creating socket to [[15019,0],1]
[cn2:00817] [[15019,1],2] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],1] via interface eth0 on socket 11
[cn2:00817] [[15019,1],2] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],1] on 192.168.1.2:49722 - 0 retries
[cn2:00817] [[15019,1],2] waiting for connect completion to [[15019,0],1] - activating send event
[cn2:00817] [[15019,1],2] tcp:send_handler called to send to peer [[15019,0],1]
[cn2:00817] [[15019,1],2] tcp:send_handler CONNECTING
[cn2:00817] [[15019,1],2]:tcp:complete_connect called for peer [[15019,0],1] on socket 11
[cn4:17680] [[15019,1],6]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17680] [[15019,1],6] RECV CONNECT ACK FROM [[15019,0],3] ON SOCKET 11
[cn4:17680] [[15019,1],6] waiting for connect ack from [[15019,0],3]
[cn4:17680] [[15019,1],6] connect ack received from [[15019,0],3]
[cn4:17680] [[15019,1],6] connect-ack recvd from [[15019,0],3]
[cn4:17680] [[15019,1],6] connect-ack header from [[15019,0],3] is okay
[cn4:17680] [[15019,1],6] waiting for connect ack from [[15019,0],3]
[cn4:17680] [[15019,1],6] connect ack received from [[15019,0],3]
[cn4:17680] [[15019,1],6] connect-ack version from [[15019,0],3] matches ours
[cn4:17680] [[15019,1],6] connect-ack [[15019,0],3] authenticated
[cn4:17680] [[15019,1],6]-[[15019,0],3] tcp_peer_connected on socket 11
[cn4:17680] [[15019,1],6]-[[15019,0],3] connected: 192.168.1.4 - 192.168.1.4 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn4:17680] [[15019,1],6]:tcp:recv:handler starting send/recv events
[cn4:17680] [[15019,1],6] tcp:set_module called for peer [[15019,0],3]
[cn4:17680] [[15019,1],6] tcp:send_handler called to send to peer [[15019,0],3]
[cn3:21268] [[15019,1],4] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 14140 BYTES FOR DEST [[15019,1],4] TAG 20
[cn3:21268] [[15019,1],4] DELIVERING TO RML
[cn3:21268] [[15019,1],4] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21268] [[15019,1],4] message received 14140 bytes from [[15019,0],2] for tag 20
[cn3:21268] [[15019,1],4] registering sync ack recvd
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 187 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 103
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 103 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 584
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 584 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 343
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 343 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 555
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 555 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 103 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 117 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 584 bytes from [[15019,0],1] for tag 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 343 bytes from [[15019,0],3] for tag 2
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 555 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14018] [[15019,1],9]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 158 BYTES ON SOCKET 9
[cn2:00817] [[15019,1],2] tcp_peer_complete_connect: sending ack to [[15019,0],1]
[cn2:00817] [[15019,1],2] SEND CONNECT ACK
[cn2:00817] fetching data from INTERNAL dstore
[cn2:00817] dstore:hash:fetch: searching for key opal.cred on proc 9574219777
[cn2:00817] dstore_hash:fetch key opal.cred for proc 9574219777 not found
[cn2:00817] [[15019,1],2] send blocking of 47 bytes to socket 11
[cn2:00817] [[15019,1],2] blocking send complete to socket 11
[cn2:00817] [[15019,1],2] tcp_peer_complete_connect: setting read event on connection to [[15019,0],1]
[cn4:17680] [[15019,1],6] tcp:send_handler SENDING TO [[15019,0],3]
[cn4:17680] [[15019,1],6] MESSAGE SEND COMPLETE TO [[15019,0],3] OF 61 BYTES ON SOCKET 11
[cn4:17680] [[15019,1],6]-[[15019,0],3] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21269] [[15019,1],5]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21269] [[15019,1],5]:tcp:recv:handler CONNECTED
[cn3:21269] [[15019,1],5] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 14140 BYTES FOR DEST [[15019,1],5] TAG 20
[cn3:21269] [[15019,1],5] DELIVERING TO RML
[cn3:21269] [[15019,1],5] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21269] [[15019,1],5] message received 14140 bytes from [[15019,0],2] for tag 20
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 397
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 397 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 319 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1462
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 1462 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 109
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 107 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 109 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 83 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 397 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 1462 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 109 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 83 bytes from [[15019,0],2] for tag 2
[cn5:14018] [[15019,1],9]:tcp:recv:handler CONNECTED
[cn5:14018] [[15019,1],9] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 14140 BYTES FOR DEST [[15019,1],9] TAG 20
[cn5:14018] [[15019,1],9] DELIVERING TO RML
[cn5:14018] [[15019,1],9] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00817] [[15019,1],2]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00817] [[15019,1],2] RECV CONNECT ACK FROM [[15019,0],1] ON SOCKET 11
[cn2:00817] [[15019,1],2] waiting for connect ack from [[15019,0],1]
[cn2:00817] [[15019,1],2] connect ack received from [[15019,0],1]
[cn2:00817] [[15019,1],2] connect-ack recvd from [[15019,0],1]
[cn2:00817] [[15019,1],2] connect-ack header from [[15019,0],1] is okay
[cn2:00817] [[15019,1],2] waiting for connect ack from [[15019,0],1]
[cn2:00817] [[15019,1],2] connect ack received from [[15019,0],1]
[cn2:00817] [[15019,1],2] connect-ack version from [[15019,0],1] matches ours
[cn2:00817] [[15019,1],2] connect-ack [[15019,0],1] authenticated
[cn2:00817] [[15019,1],2]-[[15019,0],1] tcp_peer_connected on socket 11
[cn2:00817] [[15019,1],2]-[[15019,0],1] connected: 192.168.1.2 - 192.168.1.2 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn2:00817] [[15019,1],2]:tcp:recv:handler starting send/recv events
[cn2:00817] [[15019,1],2] tcp:set_module called for peer [[15019,0],1]
[cn2:00817] [[15019,1],2] tcp:send_handler called to send to peer [[15019,0],1]
[cn2:00817] [[15019,1],2] tcp:send_handler SENDING TO [[15019,0],1]
[cn2:00817] [[15019,1],2] MESSAGE SEND COMPLETE TO [[15019,0],1] OF 61 BYTES ON SOCKET 11
[cn2:00817] [[15019,1],2]-[[15019,0],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17681] [[15019,1],7]: set_addr to uri 984285184.3;tcp://192.168.1.4:43236
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn3:21269] [[15019,1],5] registering sync ack recvd
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 115
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 115 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 97 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 369
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 369 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 175 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 875
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 875 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 2495
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 2495 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 115 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 369 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 875 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 2495 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 2019
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 2019 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14018] [[15019,1],9] message received 14140 bytes from [[15019,0],4] for tag 20
[cn2:00818] mca: base: components_open: opening grpcomm components
[cn2:00818] mca: base: components_open: found loaded component bad
[cn2:00818] mca: base: components_open: component bad open function successful
[cn2:00818] mca:base:select: Auto-selecting grpcomm components
[cn2:00818] mca:base:select:(grpcomm) Querying component [bad]
[cn4:17681] [[15019,1],7]:set_addr checking if peer [[15019,0],3] is reachable via component tcp
[cn4:17681] [[15019,1],7] oob:tcp: working peer [[15019,0],3] address tcp://192.168.1.4:43236
[cn4:17681] [[15019,1],7] PEER [[15019,0],3] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn4:17681] [[15019,1],7] PASSING ADDR 192.168.1.4 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn4:17681] [[15019,1],7]:tcp set addr for peer [[15019,0],3]
[cn4:17681] [[15019,1],7]: peer [[15019,0],3] is reachable via component tcp
[cn4:17681] [[15019,1],7] posting recv
[cn4:17681] [[15019,1],7] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn4:17681] [[15019,1],7] rml_send_msg to peer [[15019,0],3] at tag 1
[cn4:17681] [[15019,1],7] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 984285184
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285184
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21268] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 5279252480
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252480
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21268] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 9574219776
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219776
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21268] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154369
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18164154369
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 13869187072
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187072
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21268] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154368
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154368
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21268] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn3:21268] storing data in INTERNAL dstore
[cn1:14034] mca: base: components_register: registering routed components
[cn1:14034] mca: base: components_register: found loaded component binomial
[cn1:14034] mca: base: components_register: component binomial has no register or open function
[cn1:14034] mca: base: components_register: found loaded component debruijn
[cn1:14034] mca: base: components_register: component debruijn has no register or open function
[cn1:14034] mca: base: components_register: found loaded component direct
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 83 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 559
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 559 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1371
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1371 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 2019 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0] message received 83 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 559 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 1371 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 160
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 160 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1002
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1002 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21268] dstore:hash:store storing data for proc 18446744070398869505
[cn3:21268] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 18446744070398869505
[cn3:21268] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 984285185
[cn3:21268] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 984285185
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 984285185
[cn3:21268] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 984285185
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 984285185
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 984285185
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 984285185
[cn3:21268] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 984285185
[cn3:21268] fetching data from INTERNAL dstore
[cn3:21268] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 984285185
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285185
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 984285185
[cn3:21268] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 984285185
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 5279252481
[cn5:14018] [[15019,1],9] registering sync ack recvd
[cn2:00818] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn2:00818] mca:base:select:(grpcomm) Selected component [bad]
[cn2:00818] [[15019,1],3] grpcomm:base:receive start comm
[cn2:00818] [[15019,1],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn2:00818] [[15019,1],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn2:00818] [[15019,1],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn2:00818] [[15019,1],3] setting up session dir with
	tmpdir: /tmp
	host cn2
[cn4:17681] [[15019,1],7]:tcp:processing set_peer cmd for interface eth0
[cn4:17681] [[15019,1],7] SET_PEER ADDING PEER [[15019,0],3]
[cn4:17681] [[15019,1],7] set_peer: peer [[15019,0],3] is listening on interface eth0 net 192.168.1.4 port 43236
[cn4:17681] [[15019,1],7] oob:base:send to target [[15019,0],3]
[cn4:17681] [[15019,1],7] oob:tcp:send_nb to peer [[15019,0],3]:1
[cn4:17681] [[15019,1],7] tcp:send_nb to peer [[15019,0],3]
[cn4:17681] [[15019,1],7]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],3]
[cn4:17681] [[15019,1],7]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],3]:1
[cn4:17681] [[15019,1],7] routed_binomial_get([[15019,0],3]) --> [[15019,0],3]
[cn4:17681] [[15019,1],7]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],3]
[cn4:17681] [[15019,1],7] tcp:send_nb: initiating connection to [[15019,0],3]
[cn4:17681] [[15019,1],7]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],3]
[cn4:17681] [[15019,1],7] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],3] via interface eth0
[cn4:17681] [[15019,1],7] oob:tcp:peer creating socket to [[15019,0],3]
[cn1:14034] mca: base: components_register: component direct has no register or open function
[cn1:14034] mca: base: components_register: found loaded component radix
[cn1:14034] mca: base: components_register: component radix register function successful
[cn1:14034] mca: base: components_open: opening routed components
[cn1:14034] mca: base: components_open: found loaded component binomial
[cn1:14034] mca: base: components_open: found loaded component debruijn
[cn1:14034] mca: base: components_open: found loaded component direct
[cn1:14034] mca: base: components_open: found loaded component radix
[cn1:14034] mca:base:select: Auto-selecting routed components
[cn1:14034] mca:base:select:(routed) Querying component [binomial]
[cn1:14034] mca:base:select:(routed) Query of component [binomial] set priority to 70
[cn1:14034] mca:base:select:(routed) Querying component [debruijn]
[cn1:14034] mca:base:select:(routed) Query of component [debruijn] set priority to 0
[cn1:14034] mca:base:select:(routed) Querying component [direct]
[cn1:14034] mca:base:select:(routed) Query of component [direct] set priority to 0
[cn1:14034] mca:base:select:(routed) Querying component [radix]
[cn1:14034] mca:base:select:(routed) Query of component [radix] set priority to 30
[cn1:14034] mca:base:select:(routed) Selected component [binomial]
[cn1:14034] mca: base: close: unloading component debruijn
[cn1:14034] mca: base: close: unloading component direct
[cn1:14034] mca: base: close: unloading component radix
[cn1:14034] orte_routed_base_select: initializing selected component binomial
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 160 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 1002 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],2] for tag 2
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 984285184
[cn5:14018] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285184
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 18446744069414584318
[cn5:14018] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 5279252480
[cn5:14018] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252480
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 18446744069414584318
[cn5:14018] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 9574219776
[cn5:14018] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219776
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 18446744069414584318
[cn5:14018] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 13869187072
[cn5:14018] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187072
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 18446744069414584318
[cn5:14018] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 18164154368
[cn5:14018] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154368
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 18446744069414584318
[cn5:14018] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 39638990849
[cn5:14018] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39638990849
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 39638990849
[cn5:14018] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39638990849
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 18446744070398869505
[cn5:14018] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 18446744070398869505
[cn5:14018] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 984285185
[cn5:14018] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 984285185
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 984285185
[cn5:14018] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 984285185
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 984285185
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 984285185
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 984285185
[cn5:14018] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 984285185
[cn5:14018] fetching data from INTERNAL dstore
[cn5:14018] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 984285185
[cn5:14018] dstore:hash:store: stori[cn2:00818] [[15019,1],3] posting recv
[cn2:00818] [[15019,1],3] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn4:17681] [[15019,1],7] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],3] via interface eth0 on socket 11
[cn4:17681] [[15019,1],7] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],3] on 192.168.1.4:43236 - 0 retries
[cn4:17681] [[15019,1],7] waiting for connect completion to [[15019,0],3] - activating send event
[cn4:17681] [[15019,1],7] tcp:send_handler called to send to peer [[15019,0],3]
[cn4:17681] [[15019,1],7] tcp:send_handler CONNECTING
[cn4:17681] [[15019,1],7]:tcp:complete_connect called for peer [[15019,0],3] on socket 11
[cn4:17681] [[15019,1],7] tcp_peer_complete_connect: sending ack to [[15019,0],3]
[cn4:17681] [[15019,1],7] SEND CONNECT ACK
[cn4:17681] fetching data from INTERNAL dstore
[cn4:17681] dstore:hash:fetch: searching for key opal.cred on proc 31049056257
[cn4:17681] dstore_hash:fetch key opal.cred for proc 31049056257 not found
[cn4:17681] [[15019,1],7] send blocking of 47 bytes to socket 11
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 984285184
[cn3:21269] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285184
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21269] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 5279252480
[cn3:21269] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252480
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21269] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 9574219776
[cn3:21269] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219776
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21269] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22459121665
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22459121665
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 13869187072
[cn3:21269] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187072
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21269] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 18164154368
[cn3:21269] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154368
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 18446744069414584318
[cn3:21269] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 18446744070398869505
[cn3:21269] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 18446744070398869505
[cn3:21269] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 984285185
[cn3:21269] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 984285185
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 984285185
[cn3:21269] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 984285185
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 984285185
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 984285185
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 984285185
[cn3:21269] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 984285185
[cn3:21269] fetching data from INTERNAL dstore
[cn3:21269] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 984285185
[cn3:21269] dstore:hash:store: stori[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 289
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 289 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 195
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 195 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 289 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 195 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],2] for tag 2
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 984285184
[cn5:14017] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285184
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 18446744069414584318
[cn5:14017] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 5279252480
[cn5:14017] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252480
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 18446744069414584318
[cn5:14017] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 9574219776
[cn5:14017] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219776
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 18446744069414584318
[cn5:14017] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 13869187072
[cn5:14017] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187072
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 18446744069414584318
[cn5:14017] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 18164154368
[cn5:14017] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154368
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 18446744069414584318
[cn5:14017] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35344023553
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35344023553
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 18446744070398869505
[cn5:14017] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 18446744070398869505
[cn5:14017] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 984285185
[cn5:14017] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 984285185
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 984285185
[cn5:14017] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 984285185
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 984285185
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 984285185
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 984285185
[cn5:14017] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 984285185
[cn5:14017] fetching data from INTERNAL dstore
[cn5:14017] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 984285185
[cn5:14017] dstore:hash:store: stori[cn2:00818] [[15019,1],3] posting recv
[cn2:00818] [[15019,1],3] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn2:00818] [[15019,1],3] posting recv
[cn2:00818] [[15019,1],3] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn4:17681] [[15019,1],7] blocking send complete to socket 11
[cn4:17681] [[15019,1],7] tcp_peer_complete_connect: setting read event on connection to [[15019,0],3]
[cn3:21268] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5279252481
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 5279252481
[cn3:21268] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5279252481
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 5279252481
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 5279252481
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 5279252481
[cn3:21268] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5279252481
[cn3:21268] fetching data from INTERNAL dstore
[cn3:21268] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 5279252481
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252481
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 5279252481
[cn3:21268] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5279252481
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 9574219777
[cn3:21268] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9574219777
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 9574219777
[cn3:21268] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9574219777
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 9574219777
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 9574219777
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 9574219777
[cn3:21268] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9574219777
[cn3:21268] fetching data from INTERNAL dstore
[cn3:21268] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 9574219777
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219777
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 9574219777
[cn3:21268] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9574219777
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 13869187073
[cn3:21268] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13869187073
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 13869187073
[cn3:21268] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13869187073
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 13869187073
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 13869187073
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 13869187073
[cn3:21268] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13869187073
[cn3:21268] fetching data from INTERNAL dstore
[cn3:21268] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstor[cn1:14034] mca: base: components_register: registering dstore components
[cn1:14034] mca: base: components_register: found loaded component hash
[cn1:14034] mca: base: components_register: component hash register function successful
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1251
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 1251 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1462
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1462 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 1251 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 1462 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],2] for tag 2
ng key orte.hostname[OPAL_STRING] for proc 984285185
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 984285185
[cn5:14017] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 984285185
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 5279252481
[cn5:14017] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5279252481
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 5279252481
[cn5:14017] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5279252481
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 5279252481
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 5279252481
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 5279252481
[cn5:14017] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5279252481
[cn5:14017] fetching data from INTERNAL dstore
[cn5:14017] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 5279252481
[cn5:14017] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252481
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 5279252481
[cn5:14017] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5279252481
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 9574219777
[cn5:14017] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9574219777
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 9574219777
[cn5:14017] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9574219777
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 9574219777
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 9574219777
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 9574219777
[cn5:14017] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9574219777
[cn5:14017] fetching data from INTERNAL dstore
[cn5:14017] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 9574219777
[cn5:14017] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219777
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 9574219777
[cn5:14017] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9574219777
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 13869187073
[cn5:14017] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13869187073
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 13869187073
[cn5:14017] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13869187073
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 13869187073
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 13869187073
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn5:14017] storing data in NONP[cn2:00818] procdir: /tmp/openmpi-sessions-artpol@cn2_0/15019/1/3
[cn2:00818] jobdir: /tmp/openmpi-sessions-artpol@cn2_0/15019/1
[cn2:00818] top: openmpi-sessions-artpol@cn2_0
[cn2:00818] tmp: /tmp
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 13869187073
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 13869187073
[cn2:00818] [[15019,1],3] routed_binomial: init routes for proc job [15019,1]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
	daemon uri 984285184.1;tcp://192.168.1.2:49722
[cn2:00818] [[15019,1],3] [[15019,0],1] with daemon registering
[cn2:00818] [[15019,1],3] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn2:00818] [[15019,1],3] rml_send_buffer to peer [[15019,0],1] at tag 1
[cn2:00818] [[15019,1],3] registering sync waiting for ack
[cn2:00818] [[15019,1],3] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn4:17681] [[15019,1],7]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17681] [[15019,1],7] RECV CONNECT ACK FROM [[15019,0],3] ON SOCKET 11
[cn4:17681] [[15019,1],7] waiting for connect ack from [[15019,0],3]
[cn4:17681] [[15019,1],7] connect ack received from [[15019,0],3]
[cn4:17681] [[15019,1],7] connect-ack recvd from [[15019,0],3]
[cn4:17681] [[15019,1],7] connect-ack header from [[15019,0],3] is okay
[cn4:17681] [[15019,1],7] waiting for connect ack from [[15019,0],3]
[cn4:17681] [[15019,1],7] connect ack received from [[15019,0],3]
[cn4:17681] [[15019,1],7] connect-ack version from [[15019,0],3] matches ours
[cn4:17681] [[15019,1],7] connect-ack [[15019,0],3] authenticated
[cn4:17681] [[15019,1],7]-[[15019,0],3] tcp_peer_connected on socket 11
[cn4:17681] [[15019,1],7]-[[15019,0],3] connected: 192.168.1.4 - 192.168.1.4 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn4:17681] [[15019,1],7]:tcp:recv:handler starting send/recv events
[cn4:17681] [[15019,1],7] tcp:set_module called for peer [[15019,0],3]
[cn4:17681] [[15019,1],7] tcp:send_handler called to send to peer [[15019,0],3]
[cn4:17681] [[15019,1],7] tcp:send_handler SENDING TO [[15019,0],3]
[cn4:17681] [[15019,1],7] MESSAGE SEND COMPLETE TO [[15019,0],3] OF 61 BYTES ON SOCKET 11
[cn4:17681] [[15019,1],7]-[[15019,0],3] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
ng key orte.hostname[OPAL_STRING] for proc 984285185
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 984285185
[cn3:21269] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 984285185
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 5279252481
[cn3:21269] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5279252481
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 5279252481
[cn3:21269] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5279252481
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 5279252481
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 5279252481
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 5279252481
[cn3:21269] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5279252481
[cn3:21269] fetching data from INTERNAL dstore
[cn3:21269] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 5279252481
[cn3:21269] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252481
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 5279252481
[cn3:21269] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5279252481
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 9574219777
[cn3:21269] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9574219777
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 9574219777
[cn3:21269] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9574219777
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 9574219777
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 9574219777
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 9574219777
[cn3:21269] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9574219777
[cn3:21269] fetching data from INTERNAL dstore
[cn3:21269] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 9574219777
[cn3:21269] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219777
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 9574219777
[cn3:21269] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9574219777
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 13869187073
[cn3:21269] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13869187073
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 13869187073
[cn3:21269] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13869187073
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 13869187073
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 13869187073
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn3:21269] storing data in NONP[cn1:14034] mca: base: components_open: opening dstore components
[cn1:14034] mca: base: components_open: found loaded component hash
[cn1:14034] mca:dstore:select: checking available component hash
[cn1:14034] mca:dstore:select: Querying component [hash]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 109
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 109 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 273
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 273 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 109 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 273 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],2] for tag 2
ng key orte.hostname[OPAL_STRING] for proc 984285185
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 984285185
[cn5:14018] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 984285185
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 5279252481
[cn5:14018] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5279252481
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 5279252481
[cn5:14018] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5279252481
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 5279252481
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 5279252481
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 5279252481
[cn5:14018] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5279252481
[cn5:14018] fetching data from INTERNAL dstore
[cn5:14018] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 5279252481
[cn5:14018] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252481
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 5279252481
[cn5:14018] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5279252481
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 9574219777
[cn5:14018] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9574219777
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 9574219777
[cn5:14018] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9574219777
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 9574219777
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 9574219777
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 9574219777
[cn5:14018] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9574219777
[cn5:14018] fetching data from INTERNAL dstore
[cn5:14018] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 9574219777
[cn5:14018] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219777
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 9574219777
[cn5:14018] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9574219777
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 13869187073
[cn5:14018] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13869187073
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 13869187073
[cn5:14018] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13869187073
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 13869187073
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 13869187073
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn5:14018] storing data in NONP[cn2:00818] [[15019,1],3]: set_addr to uri 984285184.1;tcp://192.168.1.2:49722
[cn4:17680] [[15019,1],6]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17680] [[15019,1],6]:tcp:recv:handler CONNECTED
[cn4:17680] [[15019,1],6]:tcp:recv:handler allocate new recv msg
[cn4:17680] [[15019,1],6]:tcp:recv:handler read hdr
e:hash:store storing data for proc 13869187073
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187073
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 13869187073
[cn3:21268] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13869187073
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 18164154369
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18164154369
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18164154369
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18164154369
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 22459121665
[cn3:21268] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22459121665
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 22459121665
[cn3:21268] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22459121665
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 22459121665
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 22459121665
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 22459121665
[cn3:21268] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22459121665
[cn3:21268] fetching data from INTERNAL dstore
[cn3:21268] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 22459121665
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22459121665
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 22459121665
[cn3:21268] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22459121665
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 26754088961
[cn3:21268] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26754088961
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 26754088961
[cn3:21268] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26754088961
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 26754088961
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 26754088961
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 26754088961
[cn3:21268] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26754088961
[cn3:21268] fetching data from INTERNAL ds[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 753
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 753 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 108
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 108 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 753 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 108 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 874
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 874 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 179
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 179 BYTES FOR DEST [[15019,0],0] TAG 5
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
EER dstore
[cn5:14017] dstore:hash:store storing data for proc 13869187073
[cn5:14017] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13869187073
[cn5:14017] fetching data from INTERNAL dstore
[cn5:14017] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 13869187073
[cn5:14017] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187073
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 13869187073
[cn5:14017] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13869187073
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 18164154369
[cn5:14017] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18164154369
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 18164154369
[cn5:14017] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18164154369
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 18164154369
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 18164154369
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 18164154369
[cn5:14017] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18164154369
[cn5:14017] fetching data from INTERNAL dstore
[cn5:14017] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 18164154369
[cn5:14017] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154369
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 18164154369
[cn5:14017] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18164154369
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 22459121665
[cn5:14017] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22459121665
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 22459121665
[cn5:14017] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22459121665
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 22459121665
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 22459121665
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 22459121665
[cn5:14017] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22459121665
[cn5:14017] fetching data from INTERNAL dstore
[cn5:14017] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 22459121665
[cn5:14017] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22459121665
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 22459121665
[cn5:14017] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22459121665
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 26754088961
[cn5:14017] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26754088961
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 26754088961
[cn5:14017] dstore:hash:store: storing ke[cn2:00818] [[15019,1],3]:set_addr checking if peer [[15019,0],1] is reachable via component tcp
[cn2:00818] [[15019,1],3] oob:tcp: working peer [[15019,0],1] address tcp://192.168.1.2:49722
[cn2:00818] [[15019,1],3] PEER [[15019,0],1] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn2:00818] [[15019,1],3] PASSING ADDR 192.168.1.2 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn2:00818] [[15019,1],3]:tcp set addr for peer [[15019,0],1]
[cn2:00818] [[15019,1],3]: peer [[15019,0],1] is reachable via component tcp
[cn2:00818] [[15019,1],3] posting recv
[cn2:00818] [[15019,1],3] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn2:00818] [[15019,1],3] rml_send_msg to peer [[15019,0],1] at tag 1
[cn4:17680] [[15019,1],6]:tcp:recv:handler allocate data region of size 14140
tore
[cn3:21268] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 26754088961
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26754088961
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 26754088961
[cn3:21268] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26754088961
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 31049056257
[cn3:21268] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31049056257
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 31049056257
[cn3:21268] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31049056257
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 31049056257
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 31049056257
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 31049056257
[cn3:21268] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31049056257
[cn3:21268] fetching data from INTERNAL dstore
[cn3:21268] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 31049056257
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31049056257
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 31049056257
[cn3:21268] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31049056257
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 35344023553
[cn3:21268] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35344023553
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 35344023553
[cn3:21268] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35344023553
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 35344023553
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 35344023553
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 35344023553
[cn3:21268] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35344023553
[cn3:21268] fetching data from INTERNAL dstore
[cn3:21268] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 35344023553
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35344023553
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 35344023553
[cn3:21268] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35344023553
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 39638990849
[cn3:21268] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39638990849
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 39638990849
[cn3:21268] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39638990849
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 39638990849
[cn3:21268] dstore:hash:store: storing key opal.c[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 874 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 179 bytes from [[15019,0],3] for tag 5
[cn1:14018] [[15019,0],0] plm:base:receive processing msg
[cn1:14018] [[15019,0],0] routed_binomial: init routes for HNP job [15019,1]
[cn1:14018] [[15019,0],0] routed_binomial:callback got uri 984285185.6;tcp://192.168.1.4:51021 for job [15019,1] rank 6
[cn1:14018] [[15019,0],0] routed_binomial:callback got uri 984285185.7;tcp://192.168.1.4:49407 for job [15019,1] rank 7
[cn1:14018] [[15019,0],0] plm:base:receive done processing commands
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],2] for tag 2
EER dstore
[cn5:14018] dstore:hash:store storing data for proc 13869187073
[cn5:14018] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13869187073
[cn5:14018] fetching data from INTERNAL dstore
[cn5:14018] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 13869187073
[cn5:14018] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187073
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 13869187073
[cn5:14018] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13869187073
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 18164154369
[cn5:14018] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18164154369
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 18164154369
[cn5:14018] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18164154369
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 18164154369
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 18164154369
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 18164154369
[cn5:14018] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18164154369
[cn5:14018] fetching data from INTERNAL dstore
[cn5:14018] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 18164154369
[cn5:14018] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154369
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 18164154369
[cn5:14018] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18164154369
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 22459121665
[cn5:14018] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22459121665
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 22459121665
[cn5:14018] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22459121665
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 22459121665
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 22459121665
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 22459121665
[cn5:14018] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22459121665
[cn5:14018] fetching data from INTERNAL dstore
[cn5:14018] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 22459121665
[cn5:14018] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22459121665
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 22459121665
[cn5:14018] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22459121665
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 26754088961
[cn5:14018] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26754088961
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 26754088961
[cn5:14018] dstore:hash:store: storing ke[cn2:00818] [[15019,1],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00818] [[15019,1],3]:tcp:processing set_peer cmd for interface eth0
[cn2:00818] [[15019,1],3] SET_PEER ADDING PEER [[15019,0],1]
[cn2:00818] [[15019,1],3] set_peer: peer [[15019,0],1] is listening on interface eth0 net 192.168.1.2 port 49722
[cn2:00818] [[15019,1],3] oob:base:send to target [[15019,0],1]
[cn2:00818] [[15019,1],3] oob:tcp:send_nb to peer [[15019,0],1]:1
[cn2:00818] [[15019,1],3] tcp:send_nb to peer [[15019,0],1]
[cn2:00818] [[15019,1],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],1]
[cn2:00818] [[15019,1],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],1]:1
EER dstore
[cn3:21269] dstore:hash:store storing data for proc 13869187073
[cn3:21269] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13869187073
[cn3:21269] fetching data from INTERNAL dstore
[cn3:21269] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 13869187073
[cn3:21269] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187073
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 13869187073
[cn3:21269] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13869187073
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 22459121665
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 18164154369
[cn3:21269] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18164154369
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 18164154369
[cn3:21269] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18164154369
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 18164154369
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 18164154369
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 18164154369
[cn3:21269] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18164154369
[cn3:21269] fetching data from INTERNAL dstore
[cn3:21269] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 18164154369
[cn3:21269] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154369
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 18164154369
[cn3:21269] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18164154369
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22459121665
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22459121665
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22459121665
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 26754088961
[cn3:21269] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26754088961
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 26754088961
[cn3:21269] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26754088961
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 26754088961
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing [cn1:14034] mca: base: components_register: registering grpcomm components
[cn1:14034] mca: base: components_register: found loaded component bad
[cn1:14034] mca: base: components_register: component bad has no register or open function
[cn1:14034] mca: base: components_open: opening grpcomm components
[cn1:14034] mca: base: components_open: found loaded component bad
[cn1:14034] mca: base: components_open: component bad open function successful
[cn1:14034] mca:base:select: Auto-selecting grpcomm components
[cn1:14034] mca:base:select:(grpcomm) Querying component [bad]
[cn1:14034] mca:base:select:(grpcomm) Query of component [bad] set priority to 10
[cn1:14034] mca:base:select:(grpcomm) Selected component [bad]
[cn1:14034] [[15019,1],1] grpcomm:base:receive start comm
[cn1:14034] [[15019,1],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 30
[cn1:14034] [[15019,1],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 48
[cn1:14034] [[15019,1],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 9
[cn1:14034] [[15019,1],1] setting up session dir with
	tmpdir: /tmp
	host cn1
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1006
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 1006 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 351
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 351 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1523
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1523 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 1006 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 351 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 1523 bytes from [[15019,0],2] for tag 2
y orte.node.rank[OPAL_UINT16] for proc 26754088961
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 26754088961
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 26754088961
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 26754088961
[cn5:14017] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26754088961
[cn5:14017] fetching data from INTERNAL dstore
[cn5:14017] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 26754088961
[cn5:14017] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26754088961
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 26754088961
[cn5:14017] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26754088961
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 31049056257
[cn5:14017] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31049056257
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 31049056257
[cn5:14017] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31049056257
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 31049056257
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 31049056257
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 31049056257
[cn5:14017] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31049056257
[cn5:14017] fetching data from INTERNAL dstore
[cn5:14017] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 31049056257
[cn5:14017] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31049056257
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 31049056257
[cn5:14017] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31049056257
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 35344023553
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35344023553
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35344023553
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35344023553
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 39638990849
[cn5:14017] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39638990849
[cn2:00818] [[15019,1],3] routed_binomial_get([[15019,0],1]) --> [[15019,0],1]
[cn2:00818] [[15019,1],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],1]
[cn2:00818] [[15019,1],3] tcp:send_nb: initiating connection to [[15019,0],1]
[cn2:00818] [[15019,1],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],1]
[cn2:00818] [[15019,1],3] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],1] via interface eth0
[cn2:00818] [[15019,1],3] oob:tcp:peer creating socket to [[15019,0],1]
[cn2:00818] [[15019,1],3] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],1] via interface eth0 on socket 11
[cn2:00818] [[15019,1],3] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],1] on 192.168.1.2:49722 - 0 retries
[cn2:00818] [[15019,1],3] waiting for connect completion to [[15019,0],1] - activating send event
[cn4:17681] [[15019,1],7]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17681] [[15019,1],7]:tcp:recv:handler CONNECTED
[cn4:17681] [[15019,1],7]:tcp:recv:handler allocate new recv msg
[cn4:17681] [[15019,1],7]:tcp:recv:handler read hdr
[cn4:17681] [[15019,1],7]:tcp:recv:handler allocate data region of size 14140
puset[OPAL_STRING] for proc 39638990849
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 39638990849
[cn3:21268] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 39638990849
[cn3:21268] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39638990849
[cn3:21268] fetching data from INTERNAL dstore
[cn3:21268] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 39638990849
[cn3:21268] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39638990849
[cn3:21268] storing data in INTERNAL dstore
[cn3:21268] dstore:hash:store storing data for proc 39638990849
[cn3:21268] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39638990849
[cn3:21268] [[15019,1],4] Process bound at launch
[cn3:21268] MCW rank 4 bound to socket 0[core 0[hwt 0]]: [B/././.][./././.]
[cn3:21268] storing data in NONPEER dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn3:21268] storing data in PEER dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 18164154369
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 810
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 810 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 156
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 156 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 810 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 156 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1880
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 1880 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 517
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 517 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 429
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 429 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1881
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1881 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
y orte.node.rank[OPAL_UINT16] for proc 26754088961
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 26754088961
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 26754088961
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 26754088961
[cn5:14018] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26754088961
[cn5:14018] fetching data from INTERNAL dstore
[cn5:14018] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 26754088961
[cn5:14018] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26754088961
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 26754088961
[cn5:14018] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26754088961
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 31049056257
[cn5:14018] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31049056257
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 31049056257
[cn5:14018] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31049056257
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 31049056257
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 31049056257
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 31049056257
[cn5:14018] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31049056257
[cn5:14018] fetching data from INTERNAL dstore
[cn5:14018] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 31049056257
[cn5:14018] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31049056257
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 31049056257
[cn5:14018] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31049056257
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 39638990849
[cn5:14018] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 39638990849
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 35344023553
[cn5:14018] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35344023553
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 35344023553
[cn5:14018] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35344023553
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 35344023553
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 35344023553
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 35344023553
[cn5:14018] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35344023553
[cn5:14018] fetching data from INTERNAL dstore
[cn5:14018] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:stor[cn2:00818] [[15019,1],3] tcp:send_handler called to send to peer [[15019,0],1]
[cn2:00818] [[15019,1],3] tcp:send_handler CONNECTING
[cn2:00818] [[15019,1],3]:tcp:complete_connect called for peer [[15019,0],1] on socket 11
[cn2:00818] [[15019,1],3] tcp_peer_complete_connect: sending ack to [[15019,0],1]
[cn2:00818] [[15019,1],3] SEND CONNECT ACK
[cn2:00818] fetching data from INTERNAL dstore
[cn2:00818] dstore:hash:fetch: searching for key opal.cred on proc 13869187073
[cn2:00818] dstore_hash:fetch key opal.cred for proc 13869187073 not found
[cn2:00818] [[15019,1],3] send blocking of 47 bytes to socket 11
[cn2:00818] [[15019,1],3] blocking send complete to socket 11
[cn2:00818] [[15019,1],3] tcp_peer_complete_connect: setting read event on connection to [[15019,0],1]
[cn4:17680] [[15019,1],6]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17680] [[15019,1],6]:tcp:recv:handler CONNECTED
data for proc 26754088961
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 26754088961
[cn3:21269] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26754088961
[cn3:21269] fetching data from INTERNAL dstore
[cn3:21269] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 26754088961
[cn3:21269] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26754088961
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 26754088961
[cn3:21269] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26754088961
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 31049056257
[cn3:21269] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31049056257
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 31049056257
[cn3:21269] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31049056257
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 31049056257
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 31049056257
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 31049056257
[cn3:21269] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31049056257
[cn3:21269] fetching data from INTERNAL dstore
[cn3:21269] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 31049056257
[cn3:21269] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31049056257
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 31049056257
[cn3:21269] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31049056257
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 35344023553
[cn3:21269] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35344023553
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 35344023553
[cn3:21269] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35344023553
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 35344023553
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 35344023553
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 35344023553
[cn3:21269] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35344023553
[cn3:21269] fetching data from INTERNAL dstore
[cn3:21269] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 35344023553
[cn3:21269] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35344023553
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 35344023553
[cn3:21269] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35344023553
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 39638990849
[cn3:21269] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39638990849[cn1:14034] [[15019,1],1] posting recv
[cn1:14034] [[15019,1],1] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD]
[cn1:14034] [[15019,1],1] posting recv
[cn1:14034] [[15019,1],1] posting persistent recv on tag 48 for peer [[WILDCARD],WILDCARD]
[cn1:14034] [[15019,1],1] posting recv
[cn1:14034] [[15019,1],1] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD]
[cn1:14018] [[15019,0],0] message received 1880 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 517 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 429 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 1881 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1880
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 1880 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 975
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 975 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 83 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 464 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
e storing data for proc 35344023553
[cn5:14018] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35344023553
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 35344023553
[cn5:14018] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35344023553
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 39638990849
[cn5:14018] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39638990849
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 39638990849
[cn5:14018] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39638990849
[cn5:14018] storing data in INTERNAL dstore
[cn5:14018] dstore:hash:store storing data for proc 39638990849
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 39638990849
[cn5:14018] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 39638990849
[cn5:14018] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39638990849
[cn5:14018] [[15019,1],9] Process bound at launch
[cn5:14018] MCW rank 9 bound to socket 0[core 1[hwt 0]]: [./B/./.][./././.]
[cn5:14018] storing data in NONPEER dstore
[cn5:14018] dstore:hash:store storing data for proc 39638990849
[cn5:14018] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn5:14018] storing data in PEER dstore
[cn5:14018] dstore:hash:store storing data for proc 39638990849
[cn5:14018] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 39638990849
[cn2:00818] [[15019,1],3]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00818] [[15019,1],3] RECV CONNECT ACK FROM [[15019,0],1] ON SOCKET 11
[cn2:00818] [[15019,1],3] waiting for connect ack from [[15019,0],1]
[cn2:00818] [[15019,1],3] connect ack received from [[15019,0],1]
[cn2:00818] [[15019,1],3] connect-ack recvd from [[15019,0],1]
[cn2:00818] [[15019,1],3] connect-ack header from [[15019,0],1] is okay
[cn2:00818] [[15019,1],3] waiting for connect ack from [[15019,0],1]
[cn4:17680] [[15019,1],6] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 14140 BYTES FOR DEST [[15019,1],6] TAG 20
[cn4:17680] [[15019,1],6] DELIVERING TO RML
[cn4:17680] [[15019,1],6] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17680] [[15019,1],6] message received 14140 bytes from [[15019,0],3] for tag 20

[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 39638990849
[cn3:21269] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39638990849
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 39638990849
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 39638990849
[cn3:21269] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 39638990849
[cn3:21269] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39638990849
[cn3:21269] fetching data from INTERNAL dstore
[cn3:21269] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 39638990849
[cn3:21269] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39638990849
[cn3:21269] storing data in INTERNAL dstore
[cn3:21269] dstore:hash:store storing data for proc 39638990849
[cn3:21269] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39638990849
[cn3:21269] [[15019,1],5] Process bound at launch
[cn3:21269] MCW rank 5 bound to socket 0[core 1[hwt 0]]: [./B/./.][./././.]
[cn3:21269] storing data in NONPEER dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn3:21269] storing data in PEER dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 22459121665
[cn1:14034] procdir: /tmp/openmpi-sessions-artpol@cn1_0/15019/1/1
[cn1:14034] jobdir: /tmp/openmpi-sessions-artpol@cn1_0/15019/1
[cn1:14034] top: openmpi-sessions-artpol@cn1_0
[cn1:14034] tmp: /tmp
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key opal.job.session.dir[OPAL_STRING] for proc 5279252481
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key opal.my.session.dir[OPAL_STRING] for proc 5279252481
[cn1:14034] [[15019,1],1] routed_binomial: init routes for proc job [15019,1]
	hnp_uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
	daemon uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn1:14034] [[15019,1],1] [[15019,0],0] with daemon registering
[cn1:14034] [[15019,1],1] rml_recv_buffer_nb for peer [[WILDCARD],WILDCARD] tag 20
[cn1:14034] [[15019,1],1] rml_send_buffer to peer [[15019,0],0] at tag 1
[cn1:14034] [[15019,1],1] registering sync waiting for ack
[cn1:14034] [[15019,1],1] waiting on progress thread at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/routed/base/routed_base_fns.c:350
[cn1:14034] [[15019,1],1]: set_addr to uri 984285184.0;tcp://192.168.1.1,192.168.10.2:57056
[cn1:14034] [[15019,1],1]:set_addr checking if peer [[15019,0],0] is reachable via component tcp
[cn1:14034] [[15019,1],1] oob:tcp: working peer [[15019,0],0] address tcp://192.168.1.1,192.168.10.2:57056
[cn1:14034] [[15019,1],1] PEER [[15019,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 2 INTERFACE eth0
[cn1:14034] [[15019,1],1] PASSING ADDR 192.168.1.1 TO INTERFACE eth0 AT KERNEL INDEX 2
[cn1:14034] [[15019,1],1]:tcp set addr for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 1880 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 975 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 83 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 464 bytes from [[15019,0],2] for tag 2
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 39638990849
[cn5:14017] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39638990849
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 39638990849
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 39638990849
[cn5:14017] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 39638990849
[cn5:14017] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39638990849
[cn5:14017] fetching data from INTERNAL dstore
[cn5:14017] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 39638990849
[cn5:14017] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39638990849
[cn5:14017] storing data in INTERNAL dstore
[cn5:14017] dstore:hash:store storing data for proc 39638990849
[cn5:14017] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39638990849
[cn5:14017] [[15019,1],8] Process bound at launch
[cn5:14017] MCW rank 8 bound to socket 0[core 0[hwt 0]]: [B/././.][./././.]
[cn5:14017] storing data in NONPEER dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn5:14017] storing data in PEER dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 35344023553
[cn2:00818] [[15019,1],3] connect ack received from [[15019,0],1]
[cn2:00818] [[15019,1],3] connect-ack version from [[15019,0],1] matches ours
[cn2:00818] [[15019,1],3] connect-ack [[15019,0],1] authenticated
[cn2:00818] [[15019,1],3]-[[15019,0],1] tcp_peer_connected on socket 11
[cn2:00818] [[15019,1],3]-[[15019,0],1] connected: 192.168.1.2 - 192.168.1.2 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn2:00818] [[15019,1],3]:tcp:recv:handler starting send/recv events
[cn2:00818] [[15019,1],3] tcp:set_module called for peer [[15019,0],1]
[cn2:00818] [[15019,1],3] tcp:send_handler called to send to peer [[15019,0],1]
[cn2:00818] [[15019,1],3] tcp:send_handler SENDING TO [[15019,0],1]
[cn2:00818] [[15019,1],3] MESSAGE SEND COMPLETE TO [[15019,0],1] OF 61 BYTES ON SOCKET 11
[cn2:00818] [[15019,1],3]-[[15019,0],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17680] [[15019,1],6] registering sync ack recvd
[cn3:21268] mca: base: components_register: registering bml components
[cn3:21268] mca: base: components_register: found loaded component r2
[cn3:21268] mca: base: components_register: component r2 register function successful
[cn3:21268] mca: base: components_open: opening bml components
[cn3:21268] mca: base: components_open: found loaded component r2
[cn3:21268] mca: base: components_open: component r2 open function successful
[cn1:14034] [[15019,1],1] PEER [[15019,0],0] MAY BE REACHABLE USING MODULE AT KINDEX 3 INTERFACE eth1
[cn1:14034] [[15019,1],1] PASSING ADDR 192.168.10.2 TO INTERFACE eth1 AT KERNEL INDEX 3
[cn1:14034] [[15019,1],1]:tcp set addr for peer [[15019,0],0]
[cn1:14034] [[15019,1],1]: peer [[15019,0],0] is reachable via component tcp
[cn1:14034] [[15019,1],1] posting recv
[cn1:14034] [[15019,1],1] posting non-persistent recv on tag 20 for peer [[WILDCARD],WILDCARD]
[cn1:14034] [[15019,1],1] rml_send_msg to peer [[15019,0],0] at tag 1
[cn1:14034] [[15019,1],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14034] [[15019,1],1]:tcp:processing set_peer cmd for interface eth0
[cn1:14034] [[15019,1],1] SET_PEER ADDING PEER [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 273
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 273 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 555
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 555 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
 new connection: (23, 11) 192.168.1.1:43936
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 464 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 273 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 555 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] connection_handler: working connection (23, 11) 192.168.1.1:43936
[cn1:14018] [[15019,0],0] accept_connection: 192.168.1.1:43936
[cn1:14018] [[15019,0],0] message received 464 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 464 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 108
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 108 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 83 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 2016
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 2016 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] mca: base: components_register: registering bml components
[cn2:00817] [[15019,1],2]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00817] [[15019,1],2]:tcp:recv:handler CONNECTED
[cn2:00817] [[15019,1],2]:tcp:recv:handler allocate new recv msg
[cn2:00817] [[15019,1],2]:tcp:recv:handler read hdr
[cn4:17681] [[15019,1],7]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17681] [[15019,1],7]:tcp:recv:handler CONNECTED
[cn4:17681] [[15019,1],7] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 14140 BYTES FOR DEST [[15019,1],7] TAG 20
[cn4:17681] [[15019,1],7] DELIVERING TO RML
[cn4:17681] [[15019,1],7] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17681] [[15019,1],7] message received 14140 bytes from [[15019,0],3] for tag 20
[cn3:21269] mca: base: components_register: registering bml components
[cn3:21269] mca: base: components_register: found loaded component r2
[cn3:21269] mca: base: components_register: component r2 register function successful
[cn3:21269] mca: base: components_open: opening bml components
[cn3:21269] mca: base: components_open: found loaded component r2
[cn3:21269] mca: base: components_open: component r2 open function successful
[cn1:14034] [[15019,1],1] set_peer: peer [[15019,0],0] is listening on interface eth0 net 192.168.1.1 port 57056
[cn1:14034] [[15019,1],1]:tcp:processing set_peer cmd for interface eth1
[cn1:14034] [[15019,1],1] SET_PEER ADDING PEER [[15019,0],0]
[cn1:14034] [[15019,1],1] set_peer: peer [[15019,0],0] is listening on interface eth1 net 192.168.10.2 port 57056
[cn1:14034] [[15019,1],1] oob:base:send to target [[15019,0],0]
[cn1:14034] [[15019,1],1] oob:tcp:send_nb to peer [[15019,0],0]:1
[cn1:14034] [[15019,1],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14034] [[15019,1],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14034] [[15019,1],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:1
[cn1:14034] [[15019,1],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14034] [[15019,1],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:476] queue pending to [[15019,0],0]
[cn1:14034] [[15019,1],1] tcp:send_nb: initiating connection to [[15019,0],0]
[cn1:14034] [[15019,1],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:490] connect to [[15019,0],0]
[cn1:14034] [[15019,1],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] via interface eth0
[cn1:14034] [[15019,1],1] oob:tcp:peer creating socket to [[15019,0],0]
[cn1:14034] [[15019,1],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] via interface eth0 on socket 11
[cn1:14034] [[15019,1],1] orte_tcp_peer_try_connect: attempting to connect to proc [[15019,0],0] on 192.168.1.1:57056 - 0 retries
[cn1:14034] [[15019,1],1] waiting for connect completion to [[15019,0],0] - activating send event
[cn1:14034] [[15019,1],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14034] [[15019,1],1] tcp:send_handler CONNECTING
[cn1:14034] [[15019,1],1]:tcp:complete_connect called for peer [[15019,0],0] on socket 11
[cn1:14034] [[15019,1],1] tcp_peer_complete_connect: sending ack to [[15019,0],0]
[cn1:14034] [[15019,1],1] SEND CONNECT ACK
[cn1:14034] fetching data from INTERNAL dstore
[cn1:14034] dstore:hash:fetch: searching for key opal.cred on proc 5279252481
[cn1:14034] dstore_hash:fetch key opal.cred for proc 5279252481 not found
[cn1:14034] [[15019,1],1] send blocking of 47 bytes to socket 11
[cn1:14034] [[15019,1],1] blocking send complete to socket 11
[cn1:14034] [[15019,1],1] tcp_peer_complete_connect: setting read event on connection to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called
[cn1:14018] [[15019,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 23
[cn1:14018] [[15019,0],0] waiting for connect ack from UNKNOWN
[cn1:14018] [[15019,0],0] connect ack received from UNKNOWN
[cn1:14018] [[15019,0],0] connect-ack recvd from UNKNOWN
[cn1:14018] [[15019,0],0] mca_oob_tcp_recv_connect: connection from new peer
[cn1:14018] [[15019,0],0] connect-ack header from [[15019,1],1] is okay
[cn1:14018] [[15019,0],0] waiting for connect ack from [[15019,1],1]
[cn1:14018] [[15019,0],0] connect ack received from [[15019,1],1]
[cn1:14018] [[15019,0],0] connect-ack version from [[15019,1],1] matches ours
[cn1:14018] [[15019,0],0] connect-ack [[15019,1],1] authenticated
[cn1:14018] [[15019,0],0] tcp:peer_accept called for peer [[15019,1],1] in state UNKNOWN on socket 23
[cn1:14018] [[15019,0],0] SEND CONNECT ACK
[cn1:14018] [[15019,0],0] send blocking of 47 bytes to socket 23
[cn1:14018] [[15019,0],0] blocking send complete to socket 23
[cn1:14018] [[15019,0],0]-[[15019,1],1] tcp_peer_connected on socket 23
[cn1:14018] [[15019,0],0] routed_binomial_update: [[15019,1],1] --> [[15019,1],1]
[cn1:14018] [[15019,0],0]-[[15019,1],1] accepted: 192.168.1.1 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:14018] [[15019,0],0] message received 464 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 108 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 83 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 2016 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0] tcp:set_module called for peer [[15019,1],1]
[cn5:14018] mca: base: components_register: registering bml components
[cn5:14018] mca: base: components_register: found loaded component r2
[cn5:14018] mca: base: components_register: component r2 register function successful
[cn5:14018] mca: base: components_open: opening bml components
[cn5:14018] mca: base: components_open: found loaded component r2
[cn5:14018] mca: base: components_open: component r2 open function successful
[cn2:00817] [[15019,1],2]:tcp:recv:handler allocate data region of size 14140
[cn4:17681] [[15019,1],7] registering sync ack recvd
[cn3:21268] mca: base: components_register: registering btl components
[cn3:21268] mca: base: components_register: found loaded component openib
[cn3:21268] mca: base: components_register: component openib register function successful
[cn3:21268] mca: base: components_register: found loaded component self
[cn3:21268] mca: base: components_register: component self register function successful
[cn3:21268] mca: base: components_register: found loaded component sm
[cn3:21268] mca: base: components_register: component sm register function successful
[cn3:21268] mca: base: components_register: found loaded component tcp
[cn3:21268] mca: base: components_register: component tcp register function successful
[cn3:21268] mca: base: components_register: found loaded component usnic
[cn3:21268] mca: base: components_register: component usnic register function successful
[cn3:21268] mca: base: components_register: found loaded component vader
[cn3:21268] mca: base: components_register: component vader register function successful
[cn3:21268] mca: base: components_open: opening btl components
[cn3:21268] mca: base: components_open: found loaded component openib
[cn3:21268] mca: base: components_open: component openib open function successful
[cn3:21268] mca: base: components_open: found loaded component self
[cn3:21268] mca: base: components_open: component self open function successful
[cn3:21268] mca: base: components_open: found loaded component sm
[cn3:21268] mca: base: components_open: component sm open function successful
[cn3:21268] mca: base: components_open: found loaded component tcp
[cn3:21268] mca: base: components_open: component tcp open function successful
[cn3:21268] mca: base: components_open: found loaded component usnic
[cn3:21268] mca: base: components_open: component usnic open function successful
[cn3:21268] mca: base: components_open: found loaded component vader
[cn3:21268] mca: base: components_open: component vader open function successful
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 100
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 100 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 179
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 179 BYTES FOR DEST [[15019,0],0] TAG 5
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1033
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1033 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 2016
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 2016 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 100 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 179 bytes from [[15019,0],1] for tag 5
[cn1:14018] [[15019,0],0] plm:base:receive processing msg
[cn1:14018] [[15019,0],0] routed_binomial: init routes for HNP job [15019,1]
[cn1:14018] [[15019,0],0] routed_binomial:callback got uri 984285185.2;tcp://192.168.1.2:55643 for job [15019,1] rank 2
[cn1:14018] [[15019,0],0] routed_binomial:callback got uri 984285185.3;tcp://192.168.1.2:45161 for job [15019,1] rank 3
[cn1:14018] [[15019,0],0] plm:base:receive done processing commands
[cn1:14018] [[15019,0],0] message received 1033 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 2016 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 179
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 179 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 273
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 273 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 855
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 855 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,1],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 74
[cn1:14034] [[15019,1],1]:tcp:recv:handler called for peer [[15019,0],0]
[cn1:14034] [[15019,1],1] RECV CONNECT ACK FROM [[15019,0],0] ON SOCKET 11
[cn1:14034] [[15019,1],1] waiting for connect ack from [[15019,0],0]
[cn1:14034] [[15019,1],1] connect ack received from [[15019,0],0]
[cn1:14034] [[15019,1],1] connect-ack recvd from [[15019,0],0]
[cn1:14034] [[15019,1],1] connect-ack header from [[15019,0],0] is okay
[cn1:14034] [[15019,1],1] waiting for connect ack from [[15019,0],0]
[cn1:14034] [[15019,1],1] connect ack received from [[15019,0],0]
[cn1:14034] [[15019,1],1] connect-ack version from [[15019,0],0] matches ours
[cn1:14034] [[15019,1],1] connect-ack [[15019,0],0] authenticated
[cn1:14034] [[15019,1],1]-[[15019,0],0] tcp_peer_connected on socket 11
[cn1:14034] [[15019,1],1]-[[15019,0],0] connected: 192.168.1.1 - 192.168.1.1 nodelay 0 sndbuf 262144 rcvbuf 262144 flags 00000802
[cn1:14034] [[15019,1],1]:tcp:recv:handler starting send/recv events
[cn1:14034] [[15019,1],1] tcp:set_module called for peer [[15019,0],0]
[cn1:14034] [[15019,1],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14034] [[15019,1],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14034] [[15019,1],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 74 BYTES ON SOCKET 11
[cn1:14034] [[15019,1],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14017] mca: base: components_register: found loaded component r2
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 984285184
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285184
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17680] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 5279252480
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252480
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17680] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 9574219776
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219776
[cn3:21269] mca: base: components_register: registering btl components
[cn3:21269] mca: base: components_register: found loaded component openib
[cn3:21269] mca: base: components_register: component openib register function successful
[cn3:21269] mca: base: components_register: found loaded component self
[cn3:21269] mca: base: components_register: component self register function successful
[cn3:21269] mca: base: components_register: found loaded component sm
[cn3:21269] mca: base: components_register: component sm register function successful
[cn3:21269] mca: base: components_register: found loaded component tcp
[cn3:21269] mca: base: components_register: component tcp register function successful
[cn3:21269] mca: base: components_register: found loaded component usnic
[cn3:21269] mca: base: components_register: component usnic register function successful
[cn3:21269] mca: base: components_register: found loaded component vader
[cn3:21269] mca: base: components_register: component vader register function successful
[cn3:21269] mca: base: components_open: opening btl components
[cn3:21269] mca: base: components_open: found loaded component openib
[cn3:21269] mca: base: components_open: component openib open function successful
[cn3:21269] mca: base: components_open: found loaded component self
[cn3:21269] mca: base: components_open: component self open function successful
[cn3:21269] mca: base: components_open: found loaded component sm
[cn3:21269] mca: base: components_open: component sm open function successful
[cn3:21269] mca: base: components_open: found loaded component tcp
[cn3:21269] mca: base: components_open: component tcp open function successful
[cn3:21269] mca: base: components_open: found loaded component usnic
[cn3:21269] mca: base: components_open: component usnic open function successful
[cn3:21269] mca: base: components_open: found loaded component vader
[cn3:21269] mca: base: components_open: component vader open function successful
[cn1:14018] [[15019,0],0] message received 179 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 273 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 855 bytes from [[15019,0],2] for tag 2
[cn5:14017] mca: base: components_register: component r2 register function successful
[cn5:14017] mca: base: components_open: opening bml components
[cn2:00818] [[15019,1],3]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00818] [[15019,1],3]:tcp:recv:handler CONNECTED
[cn2:00818] [[15019,1],3]:tcp:recv:handler allocate new recv msg
[cn2:00818] [[15019,1],3]:tcp:recv:handler read hdr
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17680] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 13869187072
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187072
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17680] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26754088961
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26754088961
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 18164154368
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154368
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17680] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 18446744070398869505
[cn4:17680] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 18446744070398869505
[cn4:17680] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 984285185
[cn4:17680] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 984285185
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 984285185
[cn4:17680] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 984285185
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 984285185
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 984285185
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 984285185
[cn4:17680] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 984285185
[cn4:17680] fetching data from INTERNAL dstore
[cn4:17680] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 984285185
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285185
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 984285185
[cn4:17680] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 984285185
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 5279252481
[cn4:17680] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5279252481
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 5279252481
[cn4:17680] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5279252481
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 5279252481
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 5279252481
[cn4:17680] dstore:hash:store: storing key opal.cpuset[cn3:21268] mca: base: components_register: registering pml components
[cn3:21268] mca: base: components_register: found loaded component v
[cn3:21268] mca: base: components_register: component v register function successful
[cn3:21268] mca: base: components_register: found loaded component bfo
[cn3:21268] mca: base: components_register: component bfo register function successful
[cn3:21268] mca: base: components_register: found loaded component cm
[cn3:21268] mca: base: components_register: component cm register function successful
[cn3:21268] mca: base: components_register: found loaded component ob1
[cn3:21268] mca: base: components_register: component ob1 register function successful
[cn3:21268] mca: base: components_open: opening pml components
[cn3:21268] mca: base: components_open: found loaded component v
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 174
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 174 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 108
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 108 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 855
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 855 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 174 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 108 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 855 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 156
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 156 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 319
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 319 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] mca: base: components_open: found loaded component r2
[cn5:14017] mca: base: components_open: component r2 open function successful
[cn2:00818] [[15019,1],3]:tcp:recv:handler allocate data region of size 14140
[OPAL_STRING] for proc 5279252481
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 5279252481
[cn4:17680] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5279252481
[cn4:17680] fetching data from INTERNAL dstore
[cn4:17680] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 5279252481
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252481
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 5279252481
[cn4:17680] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5279252481
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 9574219777
[cn4:17680] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9574219777
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 9574219777
[cn4:17680] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9574219777
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 9574219777
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 9574219777
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 9574219777
[cn4:17680] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9574219777
[cn4:17680] fetching data from INTERNAL dstore
[cn4:17680] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 9574219777
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219777
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 9574219777
[cn4:17680] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9574219777
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 13869187073
[cn4:17680] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13869187073
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 13869187073
[cn4:17680] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13869187073
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 13869187073
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 13869187073
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 13869187073
[cn4:17680] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13869187073
[cn4:17680] fetching data from INTERNAL dstore
[cn4:17680] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 13869187073
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187073
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 13869187073
[cn4:17680] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13869187073
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 18164154369
[cn4:17680] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18164154369
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 18164154[cn3:21269] mca: base: components_register: registering pml components
[cn3:21269] mca: base: components_register: found loaded component v
[cn3:21269] mca: base: components_register: component v register function successful
[cn3:21269] mca: base: components_register: found loaded component bfo
[cn3:21269] mca: base: components_register: component bfo register function successful
[cn3:21269] mca: base: components_register: found loaded component cm
[cn3:21269] mca: base: components_register: component cm register function successful
[cn3:21269] mca: base: components_register: found loaded component ob1
[cn3:21269] mca: base: components_register: component ob1 register function successful
[cn3:21269] mca: base: components_open: opening pml components
[cn3:21269] mca: base: components_open: found loaded component v
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 156 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 319 bytes from [[15019,0],2] for tag 2
[cn5:14017] mca: base: components_register: registering btl components
[cn2:00817] [[15019,1],2]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00817] [[15019,1],2]:tcp:recv:handler CONNECTED
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 984285184
[cn4:17681] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285184
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17681] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 5279252480
[cn4:17681] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252480
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17681] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 9574219776
[cn4:17681] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219776
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17681] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 13869187072
[cn4:17681] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187072
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17681] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31049056257
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31049056257
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 18164154368
[cn4:17681] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154368
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 18446744069414584318
[cn4:17681] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 18446744070398869505
[cn4:17681] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 18446744070398869505
[cn4:17681] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 984285185
[cn4:17681] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 984285185
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 984285185
[cn4:17681] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 984285185
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 984285185
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 984285185
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 984285185
[cn4:17681] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 984285185
[cn4:17681] fetching data from INTERNAL dstore
[cn4:17681] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 984285185
[cn4:17681] dstore:hash:store: stori[cn3:21268] mca: base: components_open: component v open function successful
[cn3:21268] mca: base: components_open: found loaded component bfo
[cn3:21268] mca: base: components_open: component bfo open function successful
[cn3:21268] mca: base: components_open: found loaded component cm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 175
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 175 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 429
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 429 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 319
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 319 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 175 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 429 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 319 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 83
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 83 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14018] mca: base: components_register: registering btl components
[cn5:14018] mca: base: components_register: found loaded component openib
[cn2:00817] [[15019,1],2] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 14140 BYTES FOR DEST [[15019,1],2] TAG 20
[cn2:00817] [[15019,1],2] DELIVERING TO RML
[cn2:00817] [[15019,1],2] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00817] [[15019,1],2] message received 14140 bytes from [[15019,0],1] for tag 20
ng key orte.hostname[OPAL_STRING] for proc 984285185
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 984285185
[cn4:17681] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 984285185
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 5279252481
[cn4:17681] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5279252481
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 5279252481
[cn4:17681] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5279252481
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 5279252481
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 5279252481
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 5279252481
[cn4:17681] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5279252481
[cn4:17681] fetching data from INTERNAL dstore
[cn4:17681] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 5279252481
[cn4:17681] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252481
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 5279252481
[cn4:17681] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5279252481
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 9574219777
[cn4:17681] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9574219777
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 9574219777
[cn4:17681] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9574219777
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 9574219777
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 9574219777
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 9574219777
[cn4:17681] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9574219777
[cn4:17681] fetching data from INTERNAL dstore
[cn4:17681] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 9574219777
[cn4:17681] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219777
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 9574219777
[cn4:17681] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9574219777
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 13869187073
[cn4:17681] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13869187073
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 13869187073
[cn4:17681] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13869187073
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 13869187073
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 13869187073
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn4:17681] storing data in NONP[cn3:21269] mca: base: components_open: component v open function successful
[cn3:21269] mca: base: components_open: found loaded component bfo
[cn3:21269] mca: base: components_open: component bfo open function successful
[cn3:21269] mca: base: components_open: found loaded component cm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 104
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 104 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 83 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 104 bytes from [[15019,0],4] for tag 2
[cn2:00817] [[15019,1],2] registering sync ack recvd
369
[cn4:17680] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18164154369
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 18164154369
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 18164154369
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 18164154369
[cn4:17680] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18164154369
[cn4:17680] fetching data from INTERNAL dstore
[cn4:17680] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 18164154369
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154369
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 18164154369
[cn4:17680] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18164154369
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 22459121665
[cn4:17680] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22459121665
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 22459121665
[cn4:17680] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22459121665
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 22459121665
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 22459121665
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 22459121665
[cn4:17680] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22459121665
[cn4:17680] fetching data from INTERNAL dstore
[cn4:17680] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 22459121665
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22459121665
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 22459121665
[cn4:17680] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22459121665
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 26754088961
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26754088961
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26754088961
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26754088961
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 31049056257
[cn4:17680] dstore:hash:store: storing key opal.lo[cn5:14017] mca: base: components_register: found loaded component openib
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 120
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 120 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 608
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 608 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 120 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 608 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],3] for tag 2
[cn5:14018] mca: base: components_register: component openib register function successful
[cn2:00818] [[15019,1],3]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00818] [[15019,1],3]:tcp:recv:handler CONNECTED
[cn2:00818] [[15019,1],3] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 14140 BYTES FOR DEST [[15019,1],3] TAG 20
[cn2:00818] [[15019,1],3] DELIVERING TO RML
[cn2:00818] [[15019,1],3] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00818] [[15019,1],3] message received 14140 bytes from [[15019,0],1] for tag 20
[cn2:00818] [[15019,1],3] registering sync ack recvd
EER dstore
[cn4:17681] dstore:hash:store storing data for proc 13869187073
[cn4:17681] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13869187073
[cn4:17681] fetching data from INTERNAL dstore
[cn4:17681] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 13869187073
[cn4:17681] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187073
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 13869187073
[cn4:17681] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13869187073
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 18164154369
[cn4:17681] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18164154369
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 18164154369
[cn4:17681] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18164154369
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 18164154369
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 18164154369
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 18164154369
[cn4:17681] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18164154369
[cn4:17681] fetching data from INTERNAL dstore
[cn4:17681] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 18164154369
[cn4:17681] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154369
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 18164154369
[cn4:17681] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18164154369
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 22459121665
[cn4:17681] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22459121665
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 22459121665
[cn4:17681] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22459121665
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 22459121665
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 22459121665
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 22459121665
[cn4:17681] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22459121665
[cn4:17681] fetching data from INTERNAL dstore
[cn4:17681] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 22459121665
[cn4:17681] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22459121665
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 22459121665
[cn4:17681] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22459121665
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 31049056257
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 26754088961
[cn4:17681] dstore:hash:store: storing key[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 102
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 102 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 102 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],3] for tag 2
[cn5:14018] mca: base: components_register: found loaded component self
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 984285184
[cn2:00817] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285184
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00817] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 5279252480
[cn2:00817] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252480
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00817] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn2:00817] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219777
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn2:00817] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9574219777
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219776
[cn2:00817] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219776
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00817] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 13869187072
[cn2:00817] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187072
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00817] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 18164154368
[cn2:00817] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154368
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00817] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 18446744070398869505
[cn2:00817] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 18446744070398869505
[cn2:00817] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 984285185
[cn2:00817] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 984285185
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 984285185
[cn2:00817] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 984285185
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 984285185
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 984285185
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 984285185
[cn2:00817] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 984285185
[cn2:00817] fetching data from INTERNAL dstore
[cn2:00817] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 984285185
[cn2:00817] dstore:hash:store: storing kcal.rank[OPAL_UINT16] for proc 31049056257
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 31049056257
[cn4:17680] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31049056257
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 31049056257
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 31049056257
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 31049056257
[cn4:17680] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31049056257
[cn4:17680] fetching data from INTERNAL dstore
[cn4:17680] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 31049056257
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31049056257
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 31049056257
[cn4:17680] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31049056257
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 35344023553
[cn4:17680] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35344023553
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 35344023553
[cn4:17680] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35344023553
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 35344023553
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 35344023553
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 35344023553
[cn4:17680] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35344023553
[cn4:17680] fetching data from INTERNAL dstore
[cn4:17680] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 35344023553
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35344023553
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 35344023553
[cn4:17680] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35344023553
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 39638990849
[cn4:17680] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39638990849
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 39638990849
[cn4:17680] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39638990849
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 39638990849
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 39638990849
[cn4:17680] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 39638990849
[cn4:17680] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39638990849
[cn4:17680] fetching data from INTERNAL dstore
[cn4:17680] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storin[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 120
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 120 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 120 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 118
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 118 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 877
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 877 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] mca: base: components_register: component openib register function successful
ey orte.hostname[OPAL_STRING] for proc 984285185
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 984285185
[cn2:00817] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 984285185
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 5279252481
[cn2:00817] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5279252481
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 5279252481
[cn2:00817] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5279252481
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 5279252481
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 5279252481
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 5279252481
[cn2:00817] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5279252481
[cn2:00817] fetching data from INTERNAL dstore
[cn2:00817] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 5279252481
[cn2:00817] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252481
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 5279252481
[cn2:00817] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5279252481
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn2:00817] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 9574219777
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn2:00817] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9574219777
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn2:00817] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9574219777
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn2:00817] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9574219777
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 13869187073
[cn2:00817] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13869187073
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 13869187073
[cn2:00817] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13869187073
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 13869187073
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 13869187073
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 13869187073
[cn2:00817] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13869187073
[cn2:00817] fetching data from INTERNAL dstore
[cn2:00817] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn2:00817] storing data in NONPE opal.local.rank[OPAL_UINT16] for proc 26754088961
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 26754088961
[cn4:17681] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26754088961
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 26754088961
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 26754088961
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 26754088961
[cn4:17681] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26754088961
[cn4:17681] fetching data from INTERNAL dstore
[cn4:17681] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 26754088961
[cn4:17681] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26754088961
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 26754088961
[cn4:17681] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26754088961
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31049056257
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31049056257
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31049056257
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 35344023553
[cn4:17681] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35344023553
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 35344023553
[cn4:17681] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35344023553
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 35344023553
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 35344023553
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 35344023553
[cn4:17681] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35344023553
[cn4:17681] fetching data from INTERNAL dstore
[cn4:17681] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 35344023553
[cn4:17681] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35344023553
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 35344023553
[cn4:17681] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35344023553
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 39638990849
[cn4:17681] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39638990849
[cn1:14018] [[15019,0],0] message received 118 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 877 bytes from [[15019,0],3] for tag 2
[cn5:14018] mca: base: components_register: component self register function successful
ER dstore
[cn2:00817] dstore:hash:store storing data for proc 13869187073
[cn2:00817] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187073
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 13869187073
[cn2:00817] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13869187073
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 18164154369
[cn2:00817] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18164154369
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 18164154369
[cn2:00817] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18164154369
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 18164154369
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 18164154369
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 18164154369
[cn2:00817] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18164154369
[cn2:00817] fetching data from INTERNAL dstore
[cn2:00817] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 18164154369
[cn2:00817] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154369
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 18164154369
[cn2:00817] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18164154369
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 22459121665
[cn2:00817] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22459121665
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 22459121665
[cn2:00817] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22459121665
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 22459121665
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 22459121665
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 22459121665
[cn2:00817] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22459121665
[cn2:00817] fetching data from INTERNAL dstore
[cn2:00817] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 22459121665
[cn2:00817] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22459121665
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 22459121665
[cn2:00817] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22459121665
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 26754088961
[cn2:00817] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26754088961
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 26754088961
[cn2:00817] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26754088961
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 26754088961
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing dag data for proc 39638990849
[cn4:17680] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39638990849
[cn4:17680] storing data in INTERNAL dstore
[cn4:17680] dstore:hash:store storing data for proc 39638990849
[cn4:17680] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39638990849
[cn4:17680] [[15019,1],6] Process bound at launch
[cn4:17680] MCW rank 6 bound to socket 0[core 0[hwt 0]]: [B/././.][./././.]
[cn4:17680] storing data in NONPEER dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn4:17680] storing data in PEER dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 26754088961
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 100
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 100 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1880
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1880 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 100 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 1880 bytes from [[15019,0],3] for tag 2
[cn5:14018] mca: base: components_register: found loaded component sm
ta for proc 26754088961
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 26754088961
[cn2:00817] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26754088961
[cn2:00817] fetching data from INTERNAL dstore
[cn2:00817] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 26754088961
[cn2:00817] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26754088961
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 26754088961
[cn2:00817] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26754088961
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 31049056257
[cn2:00817] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31049056257
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 31049056257
[cn2:00817] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31049056257
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 31049056257
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 31049056257
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 31049056257
[cn2:00817] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31049056257
[cn2:00817] fetching data from INTERNAL dstore
[cn2:00817] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 31049056257
[cn2:00817] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31049056257
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 31049056257
[cn2:00817] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31049056257
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 35344023553
[cn2:00817] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35344023553
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 35344023553
[cn2:00817] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35344023553
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 35344023553
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 35344023553
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 35344023553
[cn2:00817] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35344023553
[cn2:00817] fetching data from INTERNAL dstore
[cn2:00817] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 35344023553
[cn2:00817] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35344023553
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 35344023553
[cn2:00817] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35344023553
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 39638990849
[cn2:00817] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39638990849
[[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 39638990849
[cn4:17681] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39638990849
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 39638990849
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 39638990849
[cn4:17681] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 39638990849
[cn4:17681] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39638990849
[cn4:17681] fetching data from INTERNAL dstore
[cn4:17681] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 39638990849
[cn4:17681] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39638990849
[cn4:17681] storing data in INTERNAL dstore
[cn4:17681] dstore:hash:store storing data for proc 39638990849
[cn4:17681] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39638990849
[cn4:17681] [[15019,1],7] Process bound at launch
[cn4:17681] MCW rank 7 bound to socket 0[core 1[hwt 0]]: [./B/./.][./././.]
[cn4:17681] storing data in NONPEER dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn4:17681] storing data in PEER dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 31049056257
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 116
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 116 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1875
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 1875 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 464 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 116 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 1875 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 464 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 225
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 225 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 464 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14018] mca: base: components_register: component sm register function successful
cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 39638990849
[cn2:00817] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39638990849
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 39638990849
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 39638990849
[cn2:00817] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 39638990849
[cn2:00817] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39638990849
[cn2:00817] fetching data from INTERNAL dstore
[cn2:00817] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 39638990849
[cn2:00817] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39638990849
[cn2:00817] storing data in INTERNAL dstore
[cn2:00817] dstore:hash:store storing data for proc 39638990849
[cn2:00817] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39638990849
[cn2:00817] [[15019,1],2] Process bound at launch
[cn2:00817] MCW rank 2 bound to socket 0[core 0[hwt 0]]: [B/././.][./././.]
[cn2:00817] storing data in NONPEER dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn2:00817] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn2:00817] storing data in PEER dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn2:00817] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 9574219777
[cn4:17680] mca: base: components_register: registering bml components
[cn4:17680] mca: base: components_register: found loaded component r2
[cn4:17680] mca: base: components_register: component r2 register function successful
[cn4:17680] mca: base: components_open: opening bml components
[cn4:17680] mca: base: components_open: found loaded component r2
[cn4:17680] mca: base: components_open: component r2 open function successful
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 225 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 464 bytes from [[15019,0],3] for tag 2
[cn5:14018] mca: base: components_register: found loaded component tcp
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 984285184
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285184
[cn4:17681] mca: base: components_register: registering bml components
[cn4:17681] mca: base: components_register: found loaded component r2
[cn4:17681] mca: base: components_register: component r2 register function successful
[cn4:17681] mca: base: components_open: opening bml components
[cn4:17681] mca: base: components_open: found loaded component r2
[cn4:17681] mca: base: components_open: component r2 open function successful
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1687
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 1687 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 2016
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 2016 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 1687 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 2016 bytes from [[15019,0],3] for tag 2
[cn5:14018] mca: base: components_register: component tcp register function successful
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00818] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 5279252480
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252480
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00818] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187073
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13869187073
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 9574219776
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219776
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00818] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187072
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187072
[cn2:00818] storing data in INTERNAL dstore
[cn4:17680] mca: base: components_register: registering btl components
[cn4:17680] mca: base: components_register: found loaded component openib
[cn4:17680] mca: base: components_register: component openib register function successful
[cn4:17680] mca: base: components_register: found loaded component self
[cn4:17680] mca: base: components_register: component self register function successful
[cn4:17680] mca: base: components_register: found loaded component sm
[cn4:17680] mca: base: components_register: component sm register function successful
[cn4:17680] mca: base: components_register: found loaded component tcp
[cn4:17680] mca: base: components_register: component tcp register function successful
[cn4:17680] mca: base: components_register: found loaded component usnic
[cn4:17680] mca: base: components_register: component usnic register function successful
[cn4:17680] mca: base: components_register: found loaded component vader
[cn4:17680] mca: base: components_register: component vader register function successful
[cn4:17680] mca: base: components_open: opening btl components
[cn4:17680] mca: base: components_open: found loaded component openib
[cn4:17680] mca: base: components_open: component openib open function successful
[cn4:17680] mca: base: components_open: found loaded component self
[cn4:17680] mca: base: components_open: component self open function successful
[cn4:17680] mca: base: components_open: found loaded component sm
[cn4:17680] mca: base: components_open: component sm open function successful
[cn4:17680] mca: base: components_open: found loaded component tcp
[cn4:17680] mca: base: components_open: component tcp open function successful
[cn4:17680] mca: base: components_open: found loaded component usnic
[cn4:17680] mca: base: components_open: component usnic open function successful
[cn4:17680] mca: base: components_open: found loaded component vader
[cn4:17680] mca: base: components_open: component vader open function successful
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 102
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 102 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 2433
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 2433 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 2016
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 2016 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 102 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 2433 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 2016 bytes from [[15019,0],3] for tag 2
[cn5:14017] mca: base: components_register: found loaded component self
[cn2:00818] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00818] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 18164154368
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154368
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 18446744069414584318
[cn2:00818] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 18446744070398869505
[cn2:00818] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 18446744070398869505
[cn2:00818] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 984285185
[cn2:00818] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 984285185
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 984285185
[cn2:00818] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 984285185
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 984285185
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 984285185
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 984285185
[cn2:00818] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 984285185
[cn2:00818] fetching data from INTERNAL dstore
[cn2:00818] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 984285185
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285185
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 984285185
[cn4:17681] mca: base: components_register: registering btl components
[cn4:17681] mca: base: components_register: found loaded component openib
[cn4:17681] mca: base: components_register: component openib register function successful
[cn4:17681] mca: base: components_register: found loaded component self
[cn4:17681] mca: base: components_register: component self register function successful
[cn4:17681] mca: base: components_register: found loaded component sm
[cn4:17681] mca: base: components_register: component sm register function successful
[cn4:17681] mca: base: components_register: found loaded component tcp
[cn4:17681] mca: base: components_register: component tcp register function successful
[cn4:17681] mca: base: components_register: found loaded component usnic
[cn4:17681] mca: base: components_register: component usnic register function successful
[cn4:17681] mca: base: components_register: found loaded component vader
[cn4:17681] mca: base: components_register: component vader register function successful
[cn4:17681] mca: base: components_open: opening btl components
[cn4:17681] mca: base: components_open: found loaded component openib
[cn4:17681] mca: base: components_open: component openib open function successful
[cn4:17681] mca: base: components_open: found loaded component self
[cn4:17681] mca: base: components_open: component self open function successful
[cn4:17681] mca: base: components_open: found loaded component sm
[cn4:17681] mca: base: components_open: component sm open function successful
[cn4:17681] mca: base: components_open: found loaded component tcp
[cn4:17681] mca: base: components_open: component tcp open function successful
[cn4:17681] mca: base: components_open: found loaded component usnic
[cn4:17681] mca: base: components_open: component usnic open function successful
[cn4:17681] mca: base: components_open: found loaded component vader
[cn4:17681] mca: base: components_open: component vader open function successful
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 103
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 103 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 2335
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 2335 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 103 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 2335 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],3] for tag 2
[cn5:14018] mca: base: components_register: found loaded component usnic
[cn2:00818] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 984285185
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 5279252481
[cn2:00818] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5279252481
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 5279252481
[cn2:00818] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5279252481
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 5279252481
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 5279252481
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 5279252481
[cn2:00818] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5279252481
[cn2:00818] fetching data from INTERNAL dstore
[cn2:00818] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 5279252481
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252481
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 5279252481
[cn2:00818] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5279252481
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 13869187073
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 9574219777
[cn2:00818] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9574219777
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 9574219777
[cn2:00818] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9574219777
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 9574219777
[cn4:17680] mca: base: components_register: registering pml components
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 119
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 119 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 2868
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 2868 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 170
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 170 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 119 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 2868 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 170 bytes from [[15019,0],3] for tag 2
[cn5:14018] mca: base: components_register: component usnic register function successful
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 9574219777
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 9574219777
[cn2:00818] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9574219777
[cn2:00818] fetching data from INTERNAL dstore
[cn2:00818] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 9574219777
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219777
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 9574219777
[cn2:00818] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9574219777
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13869187073
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13869187073
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13869187073
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 18164154369
[cn2:00818] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18164154369
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 18164154369
[cn2:00818] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18164154369
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 18164154369
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 18164154369
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn2:00818] storing data in NONPEER dstore
[cn4:17681] mca: base: components_register: registering pml components
[cn4:17681] mca: base: components_register: found loaded component v
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 103
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 103 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 3035
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 3035 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 186
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 186 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 103 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 3035 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 186 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 693
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 693 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1803
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 1803 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 187
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 187 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14018] mca: base: components_register: found loaded component vader
[cn2:00818] dstore:hash:store storing data for proc 18164154369
[cn2:00818] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18164154369
[cn2:00818] fetching data from INTERNAL dstore
[cn2:00818] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 18164154369
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154369
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 18164154369
[cn2:00818] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18164154369
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 22459121665
[cn2:00818] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22459121665
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 22459121665
[cn2:00818] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22459121665
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 22459121665
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 22459121665
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 22459121665
[cn2:00818] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22459121665
[cn2:00818] fetching data from INTERNAL dstore
[cn2:00818] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 22459121665
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22459121665
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 22459121665
[cn2:00818] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22459121665
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 26754088961
[cn2:00818] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26754088961
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 26754088961
[cn2:00818] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26754088961
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 26754088961
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 26754088961
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn4:17681] mca: base: components_register: component v register function successful
[cn4:17681] mca: base: components_register: found loaded component bfo
[cn1:14018] [[15019,0],0] message received 693 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 1803 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 187 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 328
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 328 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 2439
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 2439 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 187
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 187 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14018] mca: base: components_register: component vader register function successful
[cn5:14018] mca: base: components_open: opening btl components
[cn5:14018] mca: base: components_open: found loaded component openib
[cn5:14018] mca: base: components_open: component openib open function successful
[cn5:14018] mca: base: components_open: found loaded component self
[cn5:14018] mca: base: components_open: component self open function successful
[cn5:14018] mca: base: components_open: found loaded component sm
[cn5:14018] mca: base: components_open: component sm open function successful
[cn5:14018] mca: base: components_open: found loaded component tcp
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 26754088961
[cn2:00818] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26754088961
[cn2:00818] fetching data from INTERNAL dstore
[cn2:00818] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 26754088961
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26754088961
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 26754088961
[cn2:00818] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26754088961
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 31049056257
[cn2:00818] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31049056257
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 31049056257
[cn2:00818] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31049056257
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 31049056257
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 31049056257
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 31049056257
[cn2:00818] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31049056257
[cn2:00818] fetching data from INTERNAL dstore
[cn4:17681] mca: base: components_register: component bfo register function successful
[cn4:17681] mca: base: components_register: found loaded component cm
[cn1:14018] [[15019,0],0] message received 328 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 2439 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 187 bytes from [[15019,0],3] for tag 2
[cn5:14018] mca: base: components_open: component tcp open function successful
[cn5:14018] mca: base: components_open: found loaded component usnic
[cn5:14018] mca: base: components_open: component usnic open function successful
[cn5:14018] mca: base: components_open: found loaded component vader
[cn2:00818] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 31049056257
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31049056257
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 31049056257
[cn2:00818] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31049056257
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 35344023553
[cn2:00818] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35344023553
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 35344023553
[cn2:00818] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35344023553
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 35344023553
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 35344023553
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 35344023553
[cn2:00818] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35344023553
[cn2:00818] fetching data from INTERNAL dstore
[cn2:00818] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 35344023553
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35344023553
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 35344023553
[cn2:00818] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35344023553
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 39638990849
[cn2:00818] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39638990849
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 39638990849
[cn2:00818] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39638990849
[cn4:17681] mca: base: components_register: component cm register function successful
[cn4:17681] mca: base: components_register: found loaded component ob1
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 111
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 111 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1204
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 1204 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 111 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 1204 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],3] for tag 2
[cn5:14018] mca: base: components_open: component vader open function successful
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 39638990849
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 39638990849
[cn2:00818] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 39638990849
[cn2:00818] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39638990849
[cn2:00818] fetching data from INTERNAL dstore
[cn2:00818] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn2:00818] storing data in NONPEER dstore
[cn2:00818] dstore:hash:store storing data for proc 39638990849
[cn2:00818] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39638990849
[cn2:00818] storing data in INTERNAL dstore
[cn2:00818] dstore:hash:store storing data for proc 39638990849
[cn2:00818] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39638990849
[cn2:00818] [[15019,1],3] Process bound at launch
[cn4:17681] mca: base: components_register: component ob1 register function successful
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 118
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 118 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 149
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 149 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 158
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 158 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 118 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 149 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 158 bytes from [[15019,0],3] for tag 2
[cn5:14017] mca: base: components_register: component self register function successful
[cn2:00818] MCW rank 3 bound to socket 0[core 1[hwt 0]]: [./B/./.][./././.]
[cn2:00818] storing data in NONPEER dstore
[cn4:17681] mca: base: components_open: opening pml components
[cn4:17681] mca: base: components_open: found loaded component v
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 100
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 100 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 387
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 387 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 99
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 99 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 100 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 387 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 99 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 116
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 116 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 464 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 186
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 186 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] mca: base: components_register: found loaded component sm
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn2:00818] storing data in PEER dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 13869187073
[cn4:17680] mca: base: components_register: found loaded component v
[cn1:14018] [[15019,0],0] message received 116 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 464 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 186 bytes from [[15019,0],3] for tag 2
[cn5:14017] mca: base: components_register: component sm register function successful
[cn2:00817] mca: base: components_register: registering bml components
[cn2:00817] mca: base: components_register: found loaded component r2
[cn2:00817] mca: base: components_register: component r2 register function successful
[cn2:00817] mca: base: components_open: opening bml components
[cn2:00817] mca: base: components_open: found loaded component r2
[cn2:00817] mca: base: components_open: component r2 open function successful
[cn4:17680] mca: base: components_register: component v register function successful
[cn4:17680] mca: base: components_register: found loaded component bfo
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 464
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 464 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 187
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 187 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 464 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 187 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 2016
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 2016 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 187
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 187 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] mca: base: components_register: found loaded component tcp
[cn2:00818] mca: base: components_register: registering bml components
[cn2:00818] mca: base: components_register: found loaded component r2
[cn2:00818] mca: base: components_register: component r2 register function successful
[cn2:00818] mca: base: components_open: opening bml components
[cn2:00818] mca: base: components_open: found loaded component r2
[cn2:00818] mca: base: components_open: component r2 open function successful
[cn4:17680] mca: base: components_register: component bfo register function successful
[cn4:17680] mca: base: components_register: found loaded component cm
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 2016 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 187 bytes from [[15019,0],3] for tag 2
[cn5:14017] mca: base: components_register: component tcp register function successful
[cn2:00817] mca: base: components_register: registering btl components
[cn2:00817] mca: base: components_register: found loaded component openib
[cn2:00817] mca: base: components_register: component openib register function successful
[cn2:00817] mca: base: components_register: found loaded component self
[cn2:00817] mca: base: components_register: component self register function successful
[cn2:00817] mca: base: components_register: found loaded component sm
[cn2:00817] mca: base: components_register: component sm register function successful
[cn2:00817] mca: base: components_register: found loaded component tcp
[cn2:00817] mca: base: components_register: component tcp register function successful
[cn2:00817] mca: base: components_register: found loaded component usnic
[cn2:00817] mca: base: components_register: component usnic register function successful
[cn2:00817] mca: base: components_register: found loaded component vader
[cn2:00817] mca: base: components_register: component vader register function successful
[cn2:00817] mca: base: components_open: opening btl components
[cn2:00817] mca: base: components_open: found loaded component openib
[cn2:00817] mca: base: components_open: component openib open function successful
[cn2:00817] mca: base: components_open: found loaded component self
[cn2:00817] mca: base: components_open: component self open function successful
[cn2:00817] mca: base: components_open: found loaded component sm
[cn2:00817] mca: base: components_open: component sm open function successful
[cn2:00817] mca: base: components_open: found loaded component tcp
[cn2:00817] mca: base: components_open: component tcp open function successful
[cn2:00817] mca: base: components_open: found loaded component usnic
[cn2:00817] mca: base: components_open: component usnic open function successful
[cn2:00817] mca: base: components_open: found loaded component vader
[cn2:00817] mca: base: components_open: component vader open function successful
[cn4:17680] mca: base: components_register: component cm register function successful
[cn4:17680] mca: base: components_register: found loaded component ob1
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 103
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 103 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 2016
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 2016 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 103 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 2016 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 119
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 119 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 855
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 855 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 158
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 158 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] mca: base: components_register: found loaded component usnic
[cn2:00818] mca: base: components_register: registering btl components
[cn2:00818] mca: base: components_register: found loaded component openib
[cn2:00818] mca: base: components_register: component openib register function successful
[cn2:00818] mca: base: components_register: found loaded component self
[cn2:00818] mca: base: components_register: component self register function successful
[cn2:00818] mca: base: components_register: found loaded component sm
[cn2:00818] mca: base: components_register: component sm register function successful
[cn2:00818] mca: base: components_register: found loaded component tcp
[cn2:00818] mca: base: components_register: component tcp register function successful
[cn2:00818] mca: base: components_register: found loaded component usnic
[cn2:00818] mca: base: components_register: component usnic register function successful
[cn2:00818] mca: base: components_register: found loaded component vader
[cn2:00818] mca: base: components_register: component vader register function successful
[cn2:00818] mca: base: components_open: opening btl components
[cn2:00818] mca: base: components_open: found loaded component openib
[cn2:00818] mca: base: components_open: component openib open function successful
[cn2:00818] mca: base: components_open: found loaded component self
[cn2:00818] mca: base: components_open: component self open function successful
[cn2:00818] mca: base: components_open: found loaded component sm
[cn2:00818] mca: base: components_open: component sm open function successful
[cn2:00818] mca: base: components_open: found loaded component tcp
[cn2:00818] mca: base: components_open: component tcp open function successful
[cn2:00818] mca: base: components_open: found loaded component usnic
[cn2:00818] mca: base: components_open: component usnic open function successful
[cn2:00818] mca: base: components_open: found loaded component vader
[cn2:00818] mca: base: components_open: component vader open function successful
[cn4:17680] mca: base: components_register: component ob1 register function successful
[cn1:14018] [[15019,0],0] message received 119 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 855 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 158 bytes from [[15019,0],3] for tag 2
[cn5:14017] mca: base: components_register: component usnic register function successful
[cn2:00817] mca: base: components_register: registering pml components
[cn2:00817] mca: base: components_register: found loaded component v
[cn2:00817] mca: base: components_register: component v register function successful
[cn2:00817] mca: base: components_register: found loaded component bfo
[cn2:00817] mca: base: components_register: component bfo register function successful
[cn2:00817] mca: base: components_register: found loaded component cm
[cn2:00817] mca: base: components_register: component cm register function successful
[cn2:00817] mca: base: components_register: found loaded component ob1
[cn2:00817] mca: base: components_register: component ob1 register function successful
[cn2:00817] mca: base: components_open: opening pml components
[cn2:00817] mca: base: components_open: found loaded component v
[cn4:17680] mca: base: components_open: opening pml components
[cn4:17680] mca: base: components_open: found loaded component v
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 103
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 103 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 855
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 855 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 319
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 319 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 103 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 855 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 319 bytes from [[15019,0],3] for tag 2
[cn5:14017] mca: base: components_register: found loaded component vader
[cn2:00818] mca: base: components_register: registering pml components
[cn2:00818] mca: base: components_register: found loaded component v
[cn2:00818] mca: base: components_register: component v register function successful
[cn2:00818] mca: base: components_register: found loaded component bfo
[cn2:00818] mca: base: components_register: component bfo register function successful
[cn2:00818] mca: base: components_register: found loaded component cm
[cn2:00818] mca: base: components_register: component cm register function successful
[cn2:00818] mca: base: components_register: found loaded component ob1
[cn2:00818] mca: base: components_register: component ob1 register function successful
[cn2:00818] mca: base: components_open: opening pml components
[cn2:00818] mca: base: components_open: found loaded component v
[cn4:17681] mca: base: components_open: component v open function successful
[cn4:17681] mca: base: components_open: found loaded component bfo
[cn4:17681] mca: base: components_open: component bfo open function successful
[cn4:17681] mca: base: components_open: found loaded component cm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 119
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 119 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 319
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 319 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 319
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 319 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 119 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 319 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 319 bytes from [[15019,0],3] for tag 2
[cn5:14017] mca: base: components_register: component vader register function successful
[cn2:00817] mca: base: components_open: component v open function successful
[cn2:00817] mca: base: components_open: found loaded component bfo
[cn2:00817] mca: base: components_open: component bfo open function successful
[cn2:00817] mca: base: components_open: found loaded component cm
[cn4:17680] mca: base: components_open: component v open function successful
[cn4:17680] mca: base: components_open: found loaded component bfo
[cn4:17680] mca: base: components_open: component bfo open function successful
[cn4:17680] mca: base: components_open: found loaded component cm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 313
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 313 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 319
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 319 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 313 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 319 bytes from [[15019,0],1] for tag 2
[cn5:14017] mca: base: components_open: opening btl components
[cn5:14017] mca: base: components_open: found loaded component openib
[cn5:14017] mca: base: components_open: component openib open function successful
[cn5:14017] mca: base: components_open: found loaded component self
[cn2:00818] mca: base: components_open: component v open function successful
[cn2:00818] mca: base: components_open: found loaded component bfo
[cn2:00818] mca: base: components_open: component bfo open function successful
[cn2:00818] mca: base: components_open: found loaded component cm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 321
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 321 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 321 bytes from [[15019,0],4] for tag 2
[cn5:14017] mca: base: components_open: component self open function successful
[cn5:14017] mca: base: components_open: found loaded component sm
[cn5:14017] mca: base: components_open: component sm open function successful
[cn5:14017] mca: base: components_open: found loaded component tcp
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 409
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 409 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 409 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 170
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 170 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] mca: base: components_open: component tcp open function successful
[cn5:14017] mca: base: components_open: found loaded component usnic
[cn5:14017] mca: base: components_open: component usnic open function successful
[cn5:14017] mca: base: components_open: found loaded component vader
[cn5:14017] mca: base: components_open: component vader open function successful
[cn1:14018] [[15019,0],0] message received 170 bytes from [[15019,0],4] for tag 2
[cn5:14018] mca: base: components_register: registering pml components
[cn5:14018] mca: base: components_register: found loaded component v
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 186
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 186 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 186 bytes from [[15019,0],4] for tag 2
[cn5:14018] mca: base: components_register: component v register function successful
[cn5:14018] mca: base: components_register: found loaded component bfo
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 273
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 273 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 273 bytes from [[15019,0],4] for tag 2
[cn5:14018] mca: base: components_register: component bfo register function successful
[cn5:14018] mca: base: components_register: found loaded component cm
[cn5:14018] mca: base: components_register: component cm register function successful
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 245
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 245 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14018] mca: base: components_register: found loaded component ob1
[cn1:14018] [[15019,0],0] message received 245 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 170
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 170 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14018] mca: base: components_register: component ob1 register function successful
[cn5:14018] mca: base: components_open: opening pml components
[cn5:14018] mca: base: components_open: found loaded component v
[cn1:14018] [[15019,0],0] message received 170 bytes from [[15019,0],4] for tag 2
[cn5:14017] mca: base: components_register: registering pml components
[cn5:14017] mca: base: components_register: found loaded component v
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 186
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 186 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 186 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 187
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 187 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] mca: base: components_register: component v register function successful
[cn5:14017] mca: base: components_register: found loaded component bfo
[cn1:14018] [[15019,0],0] message received 187 bytes from [[15019,0],4] for tag 2
[cn5:14017] mca: base: components_register: component bfo register function successful
[cn5:14017] mca: base: components_register: found loaded component cm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 187
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 187 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 187 bytes from [[15019,0],4] for tag 2
[cn5:14017] mca: base: components_register: component cm register function successful
[cn5:14017] mca: base: components_register: found loaded component ob1
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 117
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 117 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 117 bytes from [[15019,0],4] for tag 2
[cn5:14017] mca: base: components_register: component ob1 register function successful
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 158
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 158 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 158 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 319
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 319 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] mca: base: components_open: opening pml components
[cn5:14017] mca: base: components_open: found loaded component v
[cn1:14018] [[15019,0],0] message received 319 bytes from [[15019,0],4] for tag 2
[cn5:14018] mca: base: components_open: component v open function successful
[cn5:14018] mca: base: components_open: found loaded component bfo
[cn5:14018] mca: base: components_open: component bfo open function successful
[cn5:14018] mca: base: components_open: found loaded component cm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 107
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 107 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 107 bytes from [[15019,0],4] for tag 2
[cn5:14017] mca: base: components_open: component v open function successful
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 97
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 97 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 97 bytes from [[15019,0],4] for tag 2
[cn5:14017] mca: base: components_open: found loaded component bfo
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 175
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 175 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 175 bytes from [[15019,0],4] for tag 2
[cn5:14017] mca: base: components_open: component bfo open function successful
[cn5:14017] mca: base: components_open: found loaded component cm
[cn1:14018] [[15019,0],0] plm:base:launch wiring up iof for job [15019,1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 172
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 172 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 172 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 172 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 455 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14017] mca: base: components_register: registering mtl components
[cn5:14017] mca: base: components_register: found loaded component psm
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 455
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 455 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 455 bytes from [[15019,0],4] for tag 2
[cn5:14017] mca: base: components_register: component psm register function successful
[cn5:14017] mca: base: components_open: opening mtl components
[cn5:14017] mca: base: components_open: found loaded component psm
[cn5:14017] mca: base: close: component psm closed
[cn5:14017] mca: base: close: unloading component psm
[cn5:14017] mca: base: close: component cm closed
[cn5:14017] mca: base: close: unloading component cm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 176
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 176 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 176 bytes from [[15019,0],4] for tag 2
[cn5:14017] mca: base: components_open: found loaded component ob1
[cn5:14017] mca: base: components_open: component ob1 open function successful
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 172
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 172 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 172 bytes from [[15019,0],4] for tag 2
[cn5:14018] mca: base: components_register: registering mtl components
[cn5:14018] mca: base: components_register: found loaded component psm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 352
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 352 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 352 bytes from [[15019,0],4] for tag 2
[cn5:14018] mca: base: components_register: component psm register function successful
[cn5:14018] mca: base: components_open: opening mtl components
[cn5:14018] mca: base: components_open: found loaded component psm
[cn5:14018] mca: base: close: component psm closed
[cn5:14018] mca: base: close: unloading component psm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 176 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 200
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 200 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 200 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14018] mca: base: close: component cm closed
[cn5:14018] mca: base: close: unloading component cm
[cn5:14018] mca: base: components_open: found loaded component ob1
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 172 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 109
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 109 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 109 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14018] mca: base: components_open: component ob1 open function successful
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 352 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 200 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 109 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 172 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 172
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 172 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00817] mca: base: components_register: registering mtl components
[cn1:14018] [[15019,0],0] message received 172 bytes from [[15019,0],1] for tag 2
[cn2:00818] mca: base: components_register: registering mtl components
[cn2:00818] mca: base: components_register: found loaded component psm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 352
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 352 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 352 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 279
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 279 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 352 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00818] mca: base: components_register: component psm register function successful
[cn2:00818] mca: base: components_open: opening mtl components
[cn2:00818] mca: base: components_open: found loaded component psm
[cn2:00818] mca: base: close: component psm closed
[cn2:00818] mca: base: close: unloading component psm
[cn1:14018] [[15019,0],0] message received 279 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00818] mca: base: close: component cm closed
[cn2:00818] mca: base: close: unloading component cm
[cn2:00818] mca: base: components_open: found loaded component ob1
[cn2:00818] mca: base: components_open: component ob1 open function successful
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 279 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00817] mca: base: components_register: found loaded component psm
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 352
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 352 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 352 bytes from [[15019,0],1] for tag 2
[cn2:00817] mca: base: components_register: component psm register function successful
[cn2:00817] mca: base: components_open: opening mtl components
[cn2:00817] mca: base: components_open: found loaded component psm
[cn2:00817] mca: base: close: component psm closed
[cn2:00817] mca: base: close: unloading component psm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 279
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 279 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 279 bytes from [[15019,0],1] for tag 2
[cn2:00817] mca: base: close: component cm closed
[cn2:00817] mca: base: close: unloading component cm
[cn2:00817] mca: base: components_open: found loaded component ob1
[cn2:00817] mca: base: components_open: component ob1 open function successful
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 352 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 279 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 172 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 352 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 279 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21268] mca: base: components_register: registering mtl components
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 172
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 172 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 172 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 352
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 352 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21269] mca: base: components_register: registering mtl components
[cn3:21269] mca: base: components_register: found loaded component psm
[cn1:14018] [[15019,0],0] message received 352 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 279
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 279 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn3:21269] mca: base: components_register: component psm register function successful
[cn3:21269] mca: base: components_open: opening mtl components
[cn3:21269] mca: base: components_open: found loaded component psm
[cn3:21269] mca: base: close: component psm closed
[cn3:21269] mca: base: close: unloading component psm
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 279 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21269] mca: base: close: component cm closed
[cn3:21269] mca: base: close: unloading component cm
[cn3:21269] mca: base: components_open: found loaded component ob1
[cn3:21269] mca: base: components_open: component ob1 open function successful
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 352 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 352
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 352 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 279 BYTES ON SOCKET 9
[cn3:21268] mca: base: components_register: found loaded component psm
[cn1:14018] [[15019,0],0] message received 352 bytes from [[15019,0],2] for tag 2
[cn3:21268] mca: base: components_register: component psm register function successful
[cn3:21268] mca: base: components_open: opening mtl components
[cn3:21268] mca: base: components_open: found loaded component psm
[cn3:21268] mca: base: close: component psm closed
[cn3:21268] mca: base: close: unloading component psm
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 279
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 279 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 279 bytes from [[15019,0],2] for tag 2
[cn3:21268] mca: base: close: component cm closed
[cn3:21268] mca: base: close: unloading component cm
[cn3:21268] mca: base: components_open: found loaded component ob1
[cn3:21268] mca: base: components_open: component ob1 open function successful
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 86
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 86 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 86 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 86 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14018] select: component v not in the include list
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 249
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 249 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn1:14018] [[15019,0],0] message received 249 bytes from [[15019,0],4] for tag 2
[cn5:14017] select: component v not in the include list
[cn5:14017] select: component bfo not in the include list
[cn5:14017] select: initializing pml component ob1
[cn5:14017] select: initializing btl component openib
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 249 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 139
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 139 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 139 bytes from [[15019,0],4] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn5:14018] select: component bfo not in the include list
[cn5:14018] select: initializing pml component ob1
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 389 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 84
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 84 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 139 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 135 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 389
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 389 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17680] mca: base: components_register: registering mtl components
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] message received 84 bytes from [[15019,0],4] for tag 2
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 389 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 84 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14018] select: initializing btl component openib
[cn4:17681] mca: base: components_register: registering mtl components
[cn4:17681] mca: base: components_register: found loaded component psm
[cn4:17681] mca: base: components_register: component psm register function successful
[cn4:17681] mca: base: components_open: opening mtl components
[cn4:17681] mca: base: components_open: found loaded component psm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 135
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 135 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 135 bytes from [[15019,0],3] for tag 2
[cn4:17681] mca: base: close: component psm closed
[cn4:17681] mca: base: close: unloading component psm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 279
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 279 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 279 bytes from [[15019,0],3] for tag 2
[cn4:17681] mca: base: close: component cm closed
[cn4:17681] mca: base: close: unloading component cm
[cn4:17681] mca: base: components_open: found loaded component ob1
[cn4:17681] mca: base: components_open: component ob1 open function successful
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 279 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 101
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 101 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 352 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 279 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 101 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 101 bytes from [[15019,0],3] for tag 2
[cn4:17680] mca: base: components_register: found loaded component psm
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 352
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 352 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 352 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 279
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 279 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17680] mca: base: components_register: component psm register function successful
[cn4:17680] mca: base: components_open: opening mtl components
[cn4:17680] mca: base: components_open: found loaded component psm
[cn4:17680] mca: base: close: component psm closed
[cn4:17680] mca: base: close: unloading component psm
[cn1:14018] [[15019,0],0] message received 279 bytes from [[15019,0],3] for tag 2
[cn4:17680] mca: base: close: component cm closed
[cn4:17680] mca: base: close: unloading component cm
[cn4:17680] mca: base: components_open: found loaded component ob1
[cn4:17680] mca: base: components_open: component ob1 open function successful
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 86
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 86 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 86 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 86 bytes from [[15019,0],1] for tag 2
[cn2:00818] select: component v not in the include list
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 249
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 249 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 249 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00817] select: component v not in the include list
[cn2:00817] select: component bfo not in the include list
[cn2:00817] select: initializing pml component ob1
[cn2:00817] select: initializing btl component openib
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 249 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 193
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 193 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 193 bytes from [[15019,0],1] for tag 2
[cn2:00818] select: component bfo not in the include list
[cn2:00818] select: initializing pml component ob1
[cn2:00818] select: initializing btl component openib
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 193 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 86
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 86 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 86 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 86 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 249 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21268] select: component v not in the include list
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 249
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 249 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 249 bytes from [[15019,0],2] for tag 2
[cn3:21269] select: component v not in the include list
[cn3:21269] select: component bfo not in the include list
[cn3:21269] select: initializing pml component ob1
[cn3:21269] select: initializing btl component openib
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 139
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 139 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 139 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] message received 139 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21268] select: component bfo not in the include list
[cn3:21268] select: initializing pml component ob1
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 84
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 84 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 84 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 84 bytes from [[15019,0],2] for tag 2
[cn3:21268] select: initializing btl component openib
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,1],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,1],0] (ORIGIN [[15019,1],0]) OF 74 BYTES FOR DEST [[15019,0],0] TAG 1
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 74 bytes from [[15019,1],0] for tag 1
[cn1:14018] [[15019,0],0] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn1:14018] [[15019,0],0] orted_recv: received sync+nidmap from local proc [[15019,1],0]
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,1],0] at tag 20
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,1],0] at tag 20
[cn1:14018] [[15019,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] oob:base:send to target [[15019,1],0]
[cn1:14018] [[15019,0],0] oob:base:send known transport for peer [[15019,1],0]
[cn1:14018] [[15019,0],0] oob:tcp:send_nb to peer [[15019,1],0]:20
[cn1:14018] [[15019,0],0] tcp:send_nb to peer [[15019,1],0]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,1],0]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,1],0]:20
[cn1:14018] [[15019,0],0] routed_binomial_get([[15019,1],0]) --> [[15019,1],0]
[cn1:14018] [[15019,0],0] tcp:send_nb: already connected to [[15019,1],0] - queueing for send
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,1],0]
[cn1:14018] [[15019,0],0] tcp:send_handler called to send to peer [[15019,1],0]
[cn1:14018] [[15019,0],0] tcp:send_handler SENDING TO [[15019,1],0]
[cn1:14018] [[15019,0],0] MESSAGE SEND COMPLETE TO [[15019,1],0] OF 14140 BYTES ON SOCKET 20
[cn1:14018] [[15019,0],0]-[[15019,1],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14033] [[15019,1],0]:tcp:recv:handler called for peer [[15019,0],0]
[cn1:14033] [[15019,1],0]:tcp:recv:handler CONNECTED
[cn1:14033] [[15019,1],0]:tcp:recv:handler allocate new recv msg
[cn1:14033] [[15019,1],0]:tcp:recv:handler read hdr
[cn1:14033] [[15019,1],0]:tcp:recv:handler allocate data region of size 14140
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 86
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 86 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 86 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 86 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 88 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17680] select: component v not in the include list
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 88
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 88 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0] message received 88 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17680] select: component bfo not in the include list
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 249
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 249 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 249 bytes from [[15019,0],3] for tag 2
[cn4:17681] select: component v not in the include list
[cn4:17681] select: component bfo not in the include list
[cn4:17681] select: initializing pml component ob1
[cn4:17681] select: initializing btl component openib
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 249 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 81
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 81 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 81 bytes from [[15019,0],3] for tag 2
[cn4:17680] select: initializing pml component ob1
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 84
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 81 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 84 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 84 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 84 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17680] select: initializing btl component openib
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 194
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 194 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 194 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 194 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 219 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5][[15019,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 219
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 219 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 219 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 215 BYTES ON SOCKET 9
[cn5][[15019,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 215
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 215 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 215 bytes from [[15019,0],4] for tag 2
[cn5:14017] [rank=8] openib: using device mlx4_0
[cn5:14017] openib BTL: rdmacm IP address not found on port
[cn5:14017] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1620
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 1620 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 1620 bytes from [[15019,0],4] for tag 2
[cn5][[15019,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn5][[15019,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn5][[15019,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn5][[15019,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn5][[15019,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn5][[15019,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn5][[15019,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn5][[15019,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x1c9afc0 for btl 0x1ca8cc0
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1620 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 216
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 216 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 216 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 216 bytes from [[15019,0],4] for tag 2
[cn5][[15019,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,1],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,1],1] (ORIGIN [[15019,1],1]) OF 74 BYTES FOR DEST [[15019,0],0] TAG 1
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 74 bytes from [[15019,1],1] for tag 1
[cn1:14018] [[15019,0],0] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SYNC_WANT_NIDMAP
[cn1:14018] [[15019,0],0] orted_recv: received sync+nidmap from local proc [[15019,1],1]
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,1],1] at tag 20
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,1],1] at tag 20
[cn1:14018] [[15019,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] oob:base:send to target [[15019,1],1]
[cn1:14018] [[15019,0],0] oob:base:send known transport for peer [[15019,1],1]
[cn1:14018] [[15019,0],0] oob:tcp:send_nb to peer [[15019,1],1]:20
[cn1:14018] [[15019,0],0] tcp:send_nb to peer [[15019,1],1]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,1],1]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,1],1]:20
[cn1:14018] [[15019,0],0] routed_binomial_get([[15019,1],1]) --> [[15019,1],1]
[cn1:14018] [[15019,0],0] tcp:send_nb: already connected to [[15019,1],1] - queueing for send
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,1],1]
[cn1:14018] [[15019,0],0] tcp:send_handler called to send to peer [[15019,1],1]
[cn1:14018] [[15019,0],0] tcp:send_handler SENDING TO [[15019,1],1]
[cn1:14018] [[15019,0],0] MESSAGE SEND COMPLETE TO [[15019,1],1] OF 14140 BYTES ON SOCKET 23
[cn1:14018] [[15019,0],0]-[[15019,1],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] plm:base:launch [15019,1] registered
[cn1:14018] [[15019,0],0] plm:base:launch job [15019,1] is not a dynamic spawn
[cn1:14018] [[15019,0],0]: Setting up debugger process table for applications
  MPIR_being_debugged = 0
  MPIR_debug_state = 1
  MPIR_partial_attach_ok = 1
  MPIR_i_am_starter = 0
  MPIR_forward_output = 0
  MPIR_proctable_size = 10
  MPIR_proctable:
    (i, host, exe, pid) = (0, cn1, /home/research/artpol/ompi_dev/app/./hellompi, 14033)
    (i, host, exe, pid) = (1, cn1, /home/research/artpol/ompi_dev/app/./hellompi, 14034)
    (i, host, exe, pid) = (2, cn2, /home/research/artpol/ompi_dev/app/./hellompi, 817)
    (i, host, exe, pid) = (3, cn2, /home/research/artpol/ompi_dev/app/./hellompi, 818)
    (i, host, exe, pid) = (4, cn3, /home/research/artpol/ompi_dev/app/./hellompi, 21268)
    (i, host, exe, pid) = (5, cn3, /home/research/artpol/ompi_dev/app/./hellompi, 21269)
    (i, host, exe, pid) = (6, cn4, /home/research/artpol/ompi_dev/app/./hellompi, 17680)
    (i, host, exe, pid) = (7, cn4, /home/research/artpol/ompi_dev/app/./hellompi, 17681)
    (i, host, exe, pid) = (8, cn5, /home/research/artpol/ompi_dev/app/./hellompi, 14017)
    (i, host, exe, pid) = (9, cn5, /home/research/artpol/ompi_dev/app/./hellompi, 14018)
MPIR_executable_path: NULL
MPIR_server_arguments: NULL
[cn1:14034] [[15019,1],1]:tcp:recv:handler called for peer [[15019,0],0]
[cn1:14034] [[15019,1],1]:tcp:recv:handler CONNECTED
[cn1:14034] [[15019,1],1]:tcp:recv:handler allocate new recv msg
[cn1:14034] [[15019,1],1]:tcp:recv:handler read hdr
[cn1:14034] [[15019,1],1]:tcp:recv:handler allocate data region of size 14140
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 194
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 194 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 194 bytes from [[15019,0],4] for tag 2
[cn5][[15019,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 194 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 219
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 219 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 219 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 219 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5][[15019,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 416
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 416 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 416 bytes from [[15019,0],4] for tag 2
[cn5][[15019,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn5][[15019,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 79
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 79 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 79 bytes from [[15019,0],4] for tag 2
[cn5:14018] [rank=9] openib: using device mlx4_0
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1234
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 1234 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 1234 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5][[15019,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn5][[15019,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn5][[15019,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn5][[15019,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn5][[15019,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn5][[15019,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x1a48fc0 for btl 0x1a56a60
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 166
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 416 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 166 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 166 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14018] openib BTL: rdmacm IP address not found on port
[cn5:14018] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 79 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1234 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 166 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 233
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 233 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 233 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 233 bytes from [[15019,0],4] for tag 2
[cn5][[15019,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 8, qpn 27535
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 455
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 455 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 455 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 393
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 393 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] storing data in PEER dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 35344023553
[cn5:14017] select: init of component openib returned success
[cn5:14017] select: initializing btl component self
[cn5:14017] select: init of component self returned success
[cn5:14017] select: initializing btl component sm
[cn1:14018] [[15019,0],0] message received 393 bytes from [[15019,0],4] for tag 2
[cn5][[15019,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 8, Port: 1, QPN: 27535
[cn5][[15019,1],8][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 455 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 393 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 216
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 216 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 216 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5][[15019,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 139
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 216 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 139 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 139 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14017] select: init of component sm returned success
[cn5:14017] select: initializing btl component tcp
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 139 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 464
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 464 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 464 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] message received 464 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14017] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn5:14017] btl: tcp: Found match: 127.0.0.1 (lo)
[cn5:14017] storing data in PEER dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 35344023553
[cn5:14017] select: init of component tcp returned success
[cn5:14017] select: initializing btl component usnic
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 79
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 79 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 79 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 79 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14017] usNIC connectivity agent initialized
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 80
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 80 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 80 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] message received 80 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14017] usNIC connectivity client initialized
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 112
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 112 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 112 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 112 bytes from [[15019,0],4] for tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 194
[cn5:14017] found 1 verbs interface
[cn5:14017] examining verbs interface: mlx4_0
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 112
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 194 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 194 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 194 BYTES ON SOCKET 9
[cn3][[15019,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 112 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 112 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 219
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 112 BYTES ON SOCKET 9
[cn5:14017] found 1 verbs interface
[cn5:14017] examining verbs interface: mlx4_0
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 219 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 219 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 219 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3][[15019,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 79
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 79 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 79 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 79 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21268] [rank=4] openib: using device mlx4_0
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 540
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1428
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1428 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] message received 1428 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn3][[15019,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn3][[15019,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn3][[15019,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn3][[15019,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn3][[15019,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn3][[15019,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn3][[15019,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 540 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0] message received 540 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 540 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 596
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 596 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 166
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 166 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14017] btl:usnic: no usNICs found
[cn5:14017] storing data in PEER dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 35344023553
[cn5:14017] select: init of component usnic returned failure
[cn5:14017] usNIC connectivity client finalized
[cn5:14017] mca: base: close: component usnic closed
[cn5:14017] mca: base: close: unloading component usnic
[cn5:14017] select: initializing btl component vader
[cn1:14018] [[15019,0],0] message received 596 bytes from [[15019,0],4] for tag 2
[cn1:14018] [[15019,0],0] message received 166 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5][[15019,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 8, qpn 27536
[cn5][[15019,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 8, Port: 1, QPN: 27536
[cn5][[15019,1],9][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn3:21268] openib BTL: rdmacm IP address not found on port
[cn3:21268] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 998
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 998 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 596 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 222
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 222 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1428 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 998 bytes from [[15019,0],4] for tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 222 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 166 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14018] storing data in PEER dstore
[cn5:14018] dstore:hash:store storing data for proc 39638990849
[cn5:14018] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 39638990849
[cn5:14018] select: init of component openib returned success
[cn5:14018] select: initializing btl component self
[cn5:14018] select: init of component self returned success
[cn5:14018] select: initializing btl component sm
[cn5:14018] select: init of component sm returned success
[cn5:14018] select: initializing btl component tcp
[cn5:14018] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn5:14018] btl: tcp: Found match: 127.0.0.1 (lo)
[cn5:14018] storing data in PEER dstore
[cn5:14018] dstore:hash:store storing data for proc 39638990849
[cn5:14018] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 39638990849
[cn5:14018] select: init of component tcp returned success
[cn5:14018] select: initializing btl component usnic
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn3][[15019,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x197af00 for btl 0x1988a60
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 222 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 998 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 628
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 628 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 628 bytes from [[15019,0],4] for tag 2
[cn5:14017] storing data in PEER dstore
[cn5:14017] dstore:hash:store storing data for proc 35344023553
[cn5:14017] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 35344023553
[cn5:14017] select: init of component vader returned success
[cn5:14017] select: init returned priority 20
[cn5:14017] selected ob1 best priority 20
[cn5:14017] select: component ob1 selected
[cn5:14017] mca: base: close: component v closed
[cn5:14017] mca: base: close: unloading component v
[cn5:14017] mca: base: close: component bfo closed
[cn5:14017] mca: base: close: unloading component bfo
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 628 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 220 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 220
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 220 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 220 bytes from [[15019,0],4] for tag 2
[cn5:14017] [[15019,1],8] running modex
[cn5:14017] [[15019,1],8] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 216
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 216 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 216 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 216 bytes from [[15019,0],2] for tag 2
[cn3][[15019,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 194
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 194 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 194 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 194 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2][[15019,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 194
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 194 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 194 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 194 bytes from [[15019,0],1] for tag 2
[cn2][[15019,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 215
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 215 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 215 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 215
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 215 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 215 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00818] [rank=3] openib: using device mlx4_0
[cn2:00818] openib BTL: rdmacm IP address not found on port
[cn2:00818] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] message received 215 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 215 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1809 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1809 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1809
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 1809 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00817] [rank=2] openib: using device mlx4_0
[cn2:00817] openib BTL: rdmacm IP address not found on port
[cn2:00817] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn1:14018] [[15019,0],0] message received 1809 bytes from [[15019,0],1] for tag 2
[cn2][[15019,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn2][[15019,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn2][[15019,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn2][[15019,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn2][[15019,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn2][[15019,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn2][[15019,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn2][[15019,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn2][[15019,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x2712fa0 for btl 0x271e460
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1809
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 1809 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 1809 bytes from [[15019,0],1] for tag 2
[cn2][[15019,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn2][[15019,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn2][[15019,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn2][[15019,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn2][[15019,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn2][[15019,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn2][[15019,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn2][[15019,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn2][[15019,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x1239fa0 for btl 0x1247ca0
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 216 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 216
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 216 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 216 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 216 bytes from [[15019,0],1] for tag 2
[cn2][[15019,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 216
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 216 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 216 bytes from [[15019,0],1] for tag 2
[cn2][[15019,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 194
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 194 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 194 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 194 bytes from [[15019,0],2] for tag 2
[cn3][[15019,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 219
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 219 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 219 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0] message received 219 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 79 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 79
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 79 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3][[15019,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] message received 79 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21269] [rank=5] openib: using device mlx4_0
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 93
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 828
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 828 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 828 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 820
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 820 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 828 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3][[15019,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn3][[15019,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn3][[15019,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn3][[15019,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 93 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 820 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 166 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 820 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 93 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 93 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn3][[15019,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn3][[15019,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn3][[15019,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn3][[15019,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x963f00 for btl 0x971a60
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14017] [[15019,1],8] grpcomm:base:modex: performing modex
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1399
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 1399 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1399 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 194
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 166
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 166 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 1399 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 166 bytes from [[15019,0],2] for tag 2
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:tcp:recv:handler called for peer [[15019,1],8]
[cn5:14014] [[15019,0],4]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]:tcp:recv:handler read hdr
[cn5:14017] fetching data from PEER dstore
[cn5:14017] dstore:hash:fetch: searching for key NULL on proc 35344023553
[cn5:14017] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 35344023553
[cn5:14017] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 35344023553
[cn5:14017] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 35344023553
[cn5:14017] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 35344023553
[cn5:14017] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 35344023553
[cn5:14017] [[15019,1],8] grpcomm:base:pack_modex: reporting 5 entries
[cn5:14017] [[15019,1],8] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn5:14017] [[15019,1],8] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn5:14017] [[15019,1],8] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn5:14017] [[15019,1],8] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn5:14017] [[15019,1],8] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn5:14017] [[15019,1],8] grpcomm:base:full:modex: executing allgather
[cn5:14017] [[15019,1],8] grpcomm:bad entering allgather
[cn5:14017] [[15019,1],8] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn5:14017] [[15019,1],8] grpcomm:base:modex: modex posted
[cn5:14014] [[15019,0],4]:tcp:recv:handler allocate data region of size 348
[cn3:21269] openib BTL: rdmacm IP address not found on port
[cn3:21269] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn5:14014] [[15019,0],4] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1441 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1441
[cn5:14014] [[15019,0],4] rml_send_msg to peer [[15019,0],0] at tag 2
[cn5:14014] [[15019,0],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14014] [[15019,0],4] oob:base:send to target [[15019,0],0]
[cn5:14014] [[15019,0],4] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 1441 BYTES FOR DEST [[15019,0],0] TAG 2
[cn5:14014] [[15019,0],4] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn5:14014] [[15019,0],4] routing direct to the HNP
[cn1:14018] [[15019,0],0] message received 1441 bytes from [[15019,0],4] for tag 2
[cn5:14014] [[15019,0],4] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn5:14014] [[15019,0],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14014] [[15019,0],4] tcp:send_handler SENDING TO [[15019,0],0]
[cn5:14014] [[15019,0],4] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 276 BYTES ON SOCKET 9
[cn5:14014] [[15019,0],4]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn5:14017] [[15019,1],8] grpcomm:bad sending collective 0 to our daemon
[cn5:14017] [[15019,1],8] rml_send_buffer to peer [[15019,0],4] at tag 30
[cn5:14017] [[15019,1],8] grpcomm:bad allgather underway
[cn5:14017] [[15019,1],8] rml_send_msg to peer [[15019,0],4] at tag 30
[cn5:14017] [[15019,1],8] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn5:14017] [[15019,1],8] oob:base:send to target [[15019,0],4]
[cn5:14017] [[15019,1],8] oob:base:send known transport for peer [[15019,0],4]
[cn5:14017] [[15019,1],8] oob:tcp:send_nb to peer [[15019,0],4]:30
[cn5:14017] [[15019,1],8] tcp:send_nb to peer [[15019,0],4]
[cn5:14017] [[15019,1],8]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],4]
[cn5:14017] [[15019,1],8]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],4]:30
[cn5:14017] [[15019,1],8] routed_binomial_get([[15019,0],4]) --> [[15019,0],4]
[cn5:14017] [[15019,1],8] tcp:send_nb: already connected to [[15019,0],4] - queueing for send
[cn5:14017] [[15019,1],8]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],4]
[cn5:14017] [[15019,1],8] tcp:send_handler called to send to peer [[15019,0],4]
[cn5:14017] [[15019,1],8] tcp:send_handler SENDING TO [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 276
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],4] (ORIGIN [[15019,0],4]) OF 276 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 276 bytes from [[15019,0],4] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn5:14017] [[15019,1],8] MESSAGE SEND COMPLETE TO [[15019,0],4] OF 348 BYTES ON SOCKET 11
[cn5:14017] [[15019,1],8]-[[15019,0],4] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 194 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 194 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 194 bytes from [[15019,0],3] for tag 2
[cn4][[15019,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 219
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 219 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 233
[cn1:14018] [[15019,0],0] message received 219 bytes from [[15019,0],3] for tag 2
[cn4][[15019,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 233 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 219 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 79
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 79 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 79 bytes from [[15019,0],3] for tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17681] [rank=7] openib: using device mlx4_0
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 393 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 233 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 233 bytes from [[15019,0],2] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 79 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1005
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1005 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 455 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 393
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 393 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn3][[15019,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 7, qpn 33162
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 1005 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] message received 393 bytes from [[15019,0],2] for tag 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4][[15019,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn4][[15019,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn4][[15019,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn4][[15019,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn4][[15019,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn3][[15019,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 7, Port: 1, QPN: 33162
[cn3][[15019,1],4][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 166
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 166 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 216 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 455
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 455 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 166 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1005 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 455 bytes from [[15019,0],2] for tag 2
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 166 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 645 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 645
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 645 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 216
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 216 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17681] openib BTL: rdmacm IP address not found on port
[cn4:17681] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn3:21268] storing data in PEER dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 18164154369
[cn3:21268] select: init of component openib returned success
[cn3:21268] select: initializing btl component self
[cn3:21268] select: init of component self returned success
[cn3:21268] select: initializing btl component sm
[cn1:14018] [[15019,0],0] message received 645 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0] message received 216 bytes from [[15019,0],2] for tag 2
[cn4][[15019,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn4][[15019,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn4][[15019,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x16edfc0 for btl 0x16f9460
[cn3][[15019,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 139 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 139
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 139 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 154 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 139 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 154
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 228 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 154 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21268] select: init of component sm returned success
[cn3:21268] select: initializing btl component tcp
[cn1:14018] [[15019,0],0] message received 154 bytes from [[15019,0],2] for tag 2
[cn3:21268] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn3:21268] btl: tcp: Found match: 127.0.0.1 (lo)
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 216
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 228
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 228 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 142 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] message received 228 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 142
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 142 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21268] storing data in PEER dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 18164154369
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 142 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 79 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 79
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 79 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21268] select: init of component tcp returned success
[cn3:21268] select: initializing btl component usnic
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 80 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 216 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 79 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 216 bytes from [[15019,0],3] for tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 112 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 80
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 80 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21268] usNIC connectivity agent initialized
[cn4][[15019,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn1:14018] [[15019,0],0] message received 80 bytes from [[15019,0],2] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21268] usNIC connectivity client initialized
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 216 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 112
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 112 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 112 bytes from [[15019,0],2] for tag 2
[cn3:21268] found 1 verbs interface
[cn3:21268] examining verbs interface: mlx4_0
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 112
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 112 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 233 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 233
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 112 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] message received 112 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 596 BYTES ON SOCKET 9
[cn3:21268] found 1 verbs interface
[cn3:21268] examining verbs interface: mlx4_0
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 596
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 596 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 998 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 596 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 487 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 83 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 998
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 998 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn3][[15019,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 7, qpn 33163
[cn3][[15019,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 7, Port: 1, QPN: 33163
[cn3][[15019,1],5][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 998 bytes from [[15019,0],2] for tag 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 487
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 393 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21269] storing data in PEER dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 22459121665
[cn3:21269] select: init of component openib returned success
[cn3:21269] select: initializing btl component self
[cn3:21269] select: init of component self returned success
[cn3:21269] select: initializing btl component sm
[cn3:21269] select: init of component sm returned success
[cn3:21269] select: initializing btl component tcp
[cn3:21269] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn3:21269] btl: tcp: Found match: 127.0.0.1 (lo)
[cn3:21269] storing data in PEER dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 22459121665
[cn3:21269] select: init of component tcp returned success
[cn3:21269] select: initializing btl component usnic
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 564 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 487 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 487 bytes from [[15019,0],2] for tag 2
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21268] btl:usnic: no usNICs found
[cn3:21268] storing data in PEER dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 18164154369
[cn3:21268] select: init of component usnic returned failure
[cn3:21268] usNIC connectivity client finalized
[cn3:21268] mca: base: close: component usnic closed
[cn3:21268] mca: base: close: unloading component usnic
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 233 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 83
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 83 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 154 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 233 bytes from [[15019,0],1] for tag 2
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 162 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] message received 83 bytes from [[15019,0],2] for tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 818 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2][[15019,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 4, qpn 37954
[cn3:21268] select: initializing btl component vader
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 340 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 393
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 393 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 393 bytes from [[15019,0],1] for tag 2
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 564
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 564 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 162
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 162 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2][[15019,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 4, Port: 1, QPN: 37954
[cn2][[15019,1],3][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn1:14018] [[15019,0],0] message received 564 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 162 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 154
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 154 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 818
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 818 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00818] storing data in PEER dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 13869187073
[cn2:00818] select: init of component openib returned success
[cn2:00818] select: initializing btl component self
[cn2:00818] select: init of component self returned success
[cn2:00818] select: initializing btl component sm
[cn2:00818] select: init of component sm returned success
[cn2:00818] select: initializing btl component tcp
[cn3:21269] usNIC connectivity client initialized
[cn3:21269] found 1 verbs interface
[cn3:21269] examining verbs interface: mlx4_0
[cn1:14018] [[15019,0],0] message received 154 bytes from [[15019,0],1] for tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 818 bytes from [[15019,0],2] for tag 2
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 233 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 214 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 340
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 340 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 112
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00818] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn2:00818] btl: tcp: Found match: 127.0.0.1 (lo)
[cn3:21268] storing data in PEER dstore
[cn3:21268] dstore:hash:store storing data for proc 18164154369
[cn3:21268] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 18164154369
[cn3:21268] select: init of component vader returned success
[cn3:21268] select: init returned priority 20
[cn3:21268] selected ob1 best priority 20
[cn3:21268] select: component ob1 selected
[cn3:21268] mca: base: close: component v closed
[cn3:21268] mca: base: close: unloading component v
[cn3:21268] mca: base: close: component bfo closed
[cn3:21268] mca: base: close: unloading component bfo
[cn3:21268] [[15019,1],4] running modex
[cn3:21268] [[15019,1],4] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 340 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 233
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 233 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00818] storing data in PEER dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 13869187073
[cn2:00818] select: init of component tcp returned success
[cn2:00818] select: initializing btl component usnic
[cn1:14018] [[15019,0],0] message received 233 bytes from [[15019,0],1] for tag 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 453 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn2][[15019,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 4, qpn 37955
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 209 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 112 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 214
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 214 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 214 bytes from [[15019,0],1] for tag 2
[cn2][[15019,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 4, Port: 1, QPN: 37955
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 453
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 453 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 112 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] message received 453 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 112 bytes from [[15019,0],2] for tag 2
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 139 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00817] storing data in PEER dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn2:00817] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 9574219777
[cn2:00817] select: init of component openib returned success
[cn2:00817] select: initializing btl component self
[cn2:00817] select: init of component self returned success
[cn2:00817] select: initializing btl component sm
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21269] found 1 verbs interface
[cn3:21269] examining verbs interface: mlx4_0
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 209
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 209 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 154 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] message received 209 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2][[15019,1],2][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 139
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 133 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 139 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn1:14018] [[15019,0],0] message received 139 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 235 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 154
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 154 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00817] select: init of component sm returned success
[cn2:00817] select: initializing btl component tcp
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 154 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 79 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 133
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 133 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 80 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1280
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00817] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn2:00817] btl: tcp: Found match: 127.0.0.1 (lo)
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 133 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 235
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 112 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 235 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00817] storing data in PEER dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn1:14018] [[15019,0],0] message received 235 bytes from [[15019,0],1] for tag 2
[cn2:00817] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 9574219777
[cn2:00817] select: init of component tcp returned success
[cn2:00817] select: initializing btl component usnic
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 79
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 79 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 79 bytes from [[15019,0],1] for tag 2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1280 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 80
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 80 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1280 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00817] usNIC connectivity agent initialized
[cn1:14018] [[15019,0],0] message received 80 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0] message received 1280 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2]:tcp:recv:handler called for peer [[15019,1],5]
[cn3:21265] [[15019,0],2]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 112
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 112 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 3119
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00817] usNIC connectivity client initialized
[cn3:21269] btl:usnic: no usNICs found
[cn3:21269] storing data in PEER dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 22459121665
[cn3:21269] select: init of component usnic returned failure
[cn3:21269] mca: base: close: component usnic closed
[cn3:21269] mca: base: close: unloading component usnic
[cn3:21269] select: initializing btl component vader
[cn3:21269] storing data in PEER dstore
[cn3:21269] dstore:hash:store storing data for proc 22459121665
[cn3:21269] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 22459121665
[cn3:21269] select: init of component vader returned success
[cn3:21269] select: init returned priority 20
[cn3:21269] selected ob1 best priority 20
[cn3:21269] select: component ob1 selected
[cn3:21269] mca: base: close: component v closed
[cn3:21269] mca: base: close: unloading component v
[cn3:21269] mca: base: close: component bfo closed
[cn3:21269] mca: base: close: unloading component bfo
[cn3:21269] [[15019,1],5] running modex
[cn3:21269] [[15019,1],5] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn3:21265] [[15019,0],2]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2]:tcp:recv:handler allocate data region of size 348
[cn1:14018] [[15019,0],0] message received 112 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 80 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00817] found 1 verbs interface
[cn2:00817] examining verbs interface: mlx4_0
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 80
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 112 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 3119 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 112 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 3119 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 80 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 3119 bytes from [[15019,0],2] for tag 2
[cn1:14018] [[15019,0],0] message received 80 bytes from [[15019,0],1] for tag 2
[cn3:21269] [[15019,1],5] grpcomm:base:modex: performing modex
[cn3:21269] fetching data from PEER dstore
[cn3:21269] dstore:hash:fetch: searching for key NULL on proc 22459121665
[cn3:21269] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 22459121665
[cn3:21269] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 22459121665
[cn3:21269] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 22459121665
[cn3:21269] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 22459121665
[cn3:21269] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 22459121665
[cn3:21269] [[15019,1],5] grpcomm:base:pack_modex: reporting 5 entries
[cn3:21269] [[15019,1],5] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn3:21269] [[15019,1],5] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn3:21269] [[15019,1],5] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn3:21269] [[15019,1],5] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn3:21269] [[15019,1],5] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn3:21269] [[15019,1],5] grpcomm:base:full:modex: executing allgather
[cn3:21269] [[15019,1],5] grpcomm:bad entering allgather
[cn3:21269] [[15019,1],5] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn3:21269] [[15019,1],5] grpcomm:base:modex: modex posted
[cn3:21269] [[15019,1],5] grpcomm:bad sending collective 0 to our daemon
[cn3:21269] [[15019,1],5] rml_send_buffer to peer [[15019,0],2] at tag 30
[cn3:21269] [[15019,1],5] grpcomm:bad allgather underway
[cn3:21269] [[15019,1],5] rml_send_msg to peer [[15019,0],2] at tag 30
[cn3:21269] [[15019,1],5] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21269] [[15019,1],5] oob:base:send to target [[15019,0],2]
[cn3:21269] [[15019,1],5] oob:base:send known transport for peer [[15019,0],2]
[cn3:21269] [[15019,1],5] oob:tcp:send_nb to peer [[15019,0],2]:30
[cn3:21269] [[15019,1],5] tcp:send_nb to peer [[15019,0],2]
[cn3:21269] [[15019,1],5]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],2]
[cn3:21269] [[15019,1],5]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],2]:30
[cn3:21269] [[15019,1],5] routed_binomial_get([[15019,0],2]) --> [[15019,0],2]
[cn3:21269] [[15019,1],5] tcp:send_nb: already connected to [[15019,0],2] - queueing for send
[cn3:21269] [[15019,1],5]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],2]
[cn3:21269] [[15019,1],5] tcp:send_handler called to send to peer [[15019,0],2]
[cn3:21269] [[15019,1],5] tcp:send_handler SENDING TO [[15019,0],2]
[cn3:21269] [[15019,1],5] MESSAGE SEND COMPLETE TO [[15019,0],2] OF 348 BYTES ON SOCKET 11
[cn3:21269] [[15019,1],5]-[[15019,0],2] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00818] usNIC connectivity client initialized
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 112
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 112 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 112 bytes from [[15019,0],1] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 112
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 112 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00818] found 1 verbs interface
[cn2:00818] examining verbs interface: mlx4_0
[cn1:14018] [[15019,0],0] message received 112 bytes from [[15019,0],1] for tag 2
[cn2:00817] found 1 verbs interface
[cn2:00817] examining verbs interface: mlx4_0
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 328
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 328 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 328 bytes from [[15019,0],1] for tag 2
[cn2:00817] btl:usnic: no usNICs found
[cn2:00817] storing data in PEER dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn2:00817] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 9574219777
[cn2:00817] select: init of component usnic returned failure
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 112
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 112 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 112 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 328 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00818] found 1 verbs interface
[cn2:00818] examining verbs interface: mlx4_0
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 112 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 194
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 194 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 194 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 194 bytes from [[15019,0],3] for tag 2
[cn4][[15019,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 215
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 215 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 215 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1807 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] message received 215 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1807
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1807 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17680] [rank=6] openib: using device mlx4_0
[cn4:17680] openib BTL: rdmacm IP address not found on port
[cn4:17680] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn1:14018] [[15019,0],0] message received 1807 bytes from [[15019,0],3] for tag 2
[cn4][[15019,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn4][[15019,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn4][[15019,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn4][[15019,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn4][[15019,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn4][[15019,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn4][[15019,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn4][[15019,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn4][[15019,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0xdf3fc0 for btl 0xdff460
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 596
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 596 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 596 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 596 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4][[15019,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 5, qpn 32319
[cn4][[15019,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 5, Port: 1, QPN: 32319
[cn4][[15019,1],7][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 998 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 998
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 216 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 998 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 998 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 216
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 216 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17681] storing data in PEER dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 31049056257
[cn4:17681] select: init of component openib returned success
[cn4:17681] select: initializing btl component self
[cn4:17681] select: init of component self returned success
[cn4:17681] select: initializing btl component sm
[cn4:17681] select: init of component sm returned success
[cn4:17681] select: initializing btl component tcp
[cn4:17681] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn4:17681] btl: tcp: Found match: 127.0.0.1 (lo)
[cn4:17681] storing data in PEER dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 31049056257
[cn4:17681] select: init of component tcp returned success
[cn4:17681] select: initializing btl component usnic
[cn1:14018] [[15019,0],0] message received 216 bytes from [[15019,0],3] for tag 2
[cn4][[15019,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 233
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 233 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 214 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 93
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 93 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] message received 93 bytes from [[15019,0],2] for tag 2
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 233 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1674
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 93 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21268] [[15019,1],4] grpcomm:base:modex: performing modex
[cn1:14018] [[15019,0],0] message received 233 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn4][[15019,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 5, qpn 32320
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 214
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 214 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 214 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 455 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1674 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4][[15019,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 5, Port: 1, QPN: 32320
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 455
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 209 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 455 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1674 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 455 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 139 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] message received 1674 bytes from [[15019,0],2] for tag 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17680] storing data in PEER dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 26754088961
[cn4:17680] select: init of component openib returned success
[cn4:17680] select: initializing btl component self
[cn4:17680] select: init of component self returned success
[cn4:17680] select: initializing btl component sm
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn3:21268] fetching data from PEER dstore
[cn3:21268] dstore:hash:fetch: searching for key NULL on proc 18164154369
[cn3:21268] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 18164154369
[cn3:21268] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 18164154369
[cn3:21268] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 18164154369
[cn3:21268] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 18164154369
[cn3:21268] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 18164154369
[cn3:21268] [[15019,1],4] grpcomm:base:pack_modex: reporting 5 entries
[cn3:21268] [[15019,1],4] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn3:21268] [[15019,1],4] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn3:21268] [[15019,1],4] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn3:21268] [[15019,1],4] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn3:21268] [[15019,1],4] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn3:21268] [[15019,1],4] grpcomm:base:full:modex: executing allgather
[cn3:21268] [[15019,1],4] grpcomm:bad entering allgather
[cn3:21268] [[15019,1],4] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn3:21268] [[15019,1],4] grpcomm:base:modex: modex posted
[cn3:21268] [[15019,1],4] grpcomm:bad sending collective 0 to our daemon
[cn3:21268] [[15019,1],4] rml_send_buffer to peer [[15019,0],2] at tag 30
[cn3:21268] [[15019,1],4] grpcomm:bad allgather underway
[cn3:21268] [[15019,1],4] rml_send_msg to peer [[15019,0],2] at tag 30
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 209
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 209 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 209 bytes from [[15019,0],3] for tag 2
[cn4][[15019,1],6][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 139
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 154 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 139 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2]:tcp:recv:handler called for peer [[15019,1],4]
[cn3:21265] [[15019,0],2]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2]:tcp:recv:handler allocate data region of size 348
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1166
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1166 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 1166 BYTES FOR DEST [[15019,0],0] TAG 2
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 139 bytes from [[15019,0],3] for tag 2
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 1166 bytes from [[15019,0],2] for tag 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 154
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 154 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 276
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 276 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 134 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17680] select: init of component sm returned success
[cn4:17680] select: initializing btl component tcp
[cn3:21268] [[15019,1],4] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21268] [[15019,1],4] oob:base:send to target [[15019,0],2]
[cn3:21268] [[15019,1],4] oob:base:send known transport for peer [[15019,0],2]
[cn3:21268] [[15019,1],4] oob:tcp:send_nb to peer [[15019,0],2]:30
[cn3:21268] [[15019,1],4] tcp:send_nb to peer [[15019,0],2]
[cn3:21268] [[15019,1],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],2]
[cn3:21268] [[15019,1],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],2]:30
[cn3:21268] [[15019,1],4] routed_binomial_get([[15019,0],2]) --> [[15019,0],2]
[cn3:21268] [[15019,1],4] tcp:send_nb: already connected to [[15019,0],2] - queueing for send
[cn3:21268] [[15019,1],4]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],2]
[cn3:21268] [[15019,1],4] tcp:send_handler called to send to peer [[15019,0],2]
[cn3:21268] [[15019,1],4] tcp:send_handler SENDING TO [[15019,0],2]
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 154 bytes from [[15019,0],3] for tag 2
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 276 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 276 bytes from [[15019,0],2] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 236 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 134
[cn4:17680] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn4:17680] btl: tcp: Found match: 127.0.0.1 (lo)
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21268] [[15019,1],4] MESSAGE SEND COMPLETE TO [[15019,0],2] OF 348 BYTES ON SOCKET 11
[cn3:21268] [[15019,1],4]-[[15019,0],2] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 79 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 80 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 134 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 112 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] message received 134 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 236
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 236 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17680] storing data in PEER dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn1:14018] [[15019,0],0] message received 236 bytes from [[15019,0],3] for tag 2
[cn4:17680] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 26754088961
[cn4:17680] select: init of component tcp returned success
[cn4:17680] select: initializing btl component usnic
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 79
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 79 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 79 bytes from [[15019,0],3] for tag 2
[cn4:17680] usNIC connectivity agent initialized
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 80
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 80 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 80 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17680] usNIC connectivity client initialized
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 80 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 112
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 112 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 112 bytes from [[15019,0],3] for tag 2
[cn4:17680] found 1 verbs interface
[cn4:17680] examining verbs interface: mlx4_0
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 80
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 112 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 80 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 80 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 112
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 112 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17681] usNIC connectivity client initialized
[cn1:14018] [[15019,0],0] message received 112 bytes from [[15019,0],3] for tag 2
[cn4:17681] found 1 verbs interface
[cn4:17681] examining verbs interface: mlx4_0
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 112
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 112 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 112 bytes from [[15019,0],3] for tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 112 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4126
[cn4:17680] found 1 verbs interface
[cn4:17680] examining verbs interface: mlx4_0
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 112
[cn2:00814] [[15019,0],1]:tcp:recv:handler called for peer [[15019,1],2]
[cn2:00814] [[15019,0],1]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1]:tcp:recv:handler allocate data region of size 348
[cn2:00814] [[15019,0],1]:tcp:recv:handler called for peer [[15019,1],3]
[cn2:00814] [[15019,0],1]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1]:tcp:recv:handler allocate data region of size 348
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 4126 BYTES FOR DEST [[15019,0],0] TAG 2
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0] message received 4126 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00818] btl:usnic: no usNICs found
[cn2:00818] storing data in PEER dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 13869187073
[cn2:00818] select: init of component usnic returned failure
[cn2:00818] mca: base: close: component usnic closed
[cn2:00818] mca: base: close: unloading component usnic
[cn2:00818] select: initializing btl component vader
[cn2:00818] storing data in PEER dstore
[cn2:00818] dstore:hash:store storing data for proc 13869187073
[cn2:00818] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 13869187073
[cn2:00818] select: init of component vader returned success
[cn2:00818] select: init returned priority 20
[cn2:00818] selected ob1 best priority 20
[cn2:00818] select: component ob1 selected
[cn2:00818] mca: base: close: component v closed
[cn2:00818] mca: base: close: unloading component v
[cn2:00818] mca: base: close: component bfo closed
[cn2:00818] mca: base: close: unloading component bfo
[cn2:00818] [[15019,1],3] running modex
[cn2:00818] [[15019,1],3] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn2:00818] [[15019,1],3] grpcomm:base:modex: performing modex
[cn2:00818] fetching data from PEER dstore
[cn2:00818] dstore:hash:fetch: searching for key NULL on proc 13869187073
[cn2:00818] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 13869187073
[cn2:00818] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 13869187073
[cn2:00818] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 13869187073
[cn2:00818] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 13869187073
[cn2:00818] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 13869187073
[cn2:00818] [[15019,1],3] grpcomm:base:pack_modex: reporting 5 entries
[cn2:00818] [[15019,1],3] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn2:00818] [[15019,1],3] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn2:00818] [[15019,1],3] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn2:00818] [[15019,1],3] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn2:00818] [[15019,1],3] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn2:00818] [[15019,1],3] grpcomm:base:full:modex: executing allgather
[cn2:00818] [[15019,1],3] grpcomm:bad entering allgather
[cn2:00818] [[15019,1],3] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn2:00818] [[15019,1],3] grpcomm:base:modex: modex posted
[cn2:00818] [[15019,1],3] grpcomm:bad sending collective 0 to our daemon
[cn2:00818] [[15019,1],3] rml_send_buffer to peer [[15019,0],1] at tag 30
[cn2:00818] [[15019,1],3] grpcomm:bad allgather underway
[cn2:00818] [[15019,1],3] rml_send_msg to peer [[15019,0],1] at tag 30
[cn2:00818] [[15019,1],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00818] [[15019,1],3] oob:base:send to target [[15019,0],1]
[cn2:00818] [[15019,1],3] oob:base:send known transport for peer [[15019,0],1]
[cn2:00818] [[15019,1],3] oob:tcp:send_nb to peer [[15019,0],1]:30
[cn2:00818] [[15019,1],3] tcp:send_nb to peer [[15019,0],1]
[cn2:00818] [[15019,1],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],1]
[cn2:00818] [[15019,1],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],1]:30
[cn2:00818] [[15019,1],3] routed_binomial_get([[15019,0],1]) --> [[15019,0],1]
[cn2:00818] [[15019,1],3] tcp:send_nb: already connected to [[15019,0],1] - queueing for send
[cn2:00818] [[15019,1],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],1]
[cn2:00818] [[15019,1],3] tcp:send_handler called to send to peer [[15019,0],1]
[cn2:00818] [[15019,1],3] tcp:send_handler SENDING TO [[15019,0],1]
[cn[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 4109
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 112 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 112 bytes from [[15019,0],3] for tag 2
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4126 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 330
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 330 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn4:17681] found 1 verbs interface
[cn4:17681] examining verbs interface: mlx4_0
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 330 bytes from [[15019,0],3] for tag 2
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 112 BYTES ON SOCKET 9
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17680] btl:usnic: no usNICs found
[cn4:17680] storing data in PEER dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 26754088961
[cn4:17680] select: init of component usnic returned failure
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 4109 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 273 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 4109 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 4109 bytes from [[15019,0],1] for tag 2
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 330 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 273
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 273 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 78
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 78 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 78 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn2:00817] usNIC connectivity client finalized
[cn2:00817] mca: base: close: component usnic closed
[cn2:00817] mca: base: close: unloading component usnic
[cn2:00817] select: initializing btl component vader
[cn2:00817] storing data in PEER dstore
[cn2:00817] dstore:hash:store storing data for proc 9574219777
[cn2:00817] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 9574219777
[cn2:00817] select: init of component vader returned success
[cn2:00817] select: init returned priority 20
[cn2:00817] selected ob1 best priority 20
[cn2:00817] select: component ob1 selected
[cn2:00817] mca: base: close: component v closed
[cn2:00817] mca: base: close: unloading component v
[cn2:00817] mca: base: close: component bfo closed
[cn2:00817] mca: base: close: unloading component bfo
[cn2:00817] [[15019,1],2] running modex
[cn2:00817] [[15019,1],2] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn2:00817] [[15019,1],2] grpcomm:base:modex: performing modex
[cn2:00817] fetching data from PEER dstore
[cn2:00817] dstore:hash:fetch: searching for key NULL on proc 9574219777
[cn2:00817] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 9574219777
[cn2:00817] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 9574219777
[cn2:00817] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 9574219777
[cn2:00817] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 9574219777
[cn2:00817] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 9574219777
[cn2:00817] [[15019,1],2] grpcomm:base:pack_modex: reporting 5 entries
[cn2:00817] [[15019,1],2] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn2:00817] [[15019,1],2] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn2:00817] [[15019,1],2] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn2:00817] [[15019,1],2] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn2:00817] [[15019,1],2] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn2:00817] [[15019,1],2] grpcomm:base:full:modex: executing allgather
[cn2:00817] [[15019,1],2] grpcomm:bad entering allgather
[cn2:00817] [[15019,1],2] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn2:00817] [[15019,1],2] grpcomm:base:modex: modex posted
[cn2:00817] [[15019,1],2] grpcomm:bad sending collective 0 to our daemon
[cn2:00817] [[15019,1],2] rml_send_buffer to peer [[15019,0],1] at tag 30
[cn2:00817] [[15019,1],2] grpcomm:bad allgather underway
[cn2:00817] [[15019,1],2] rml_send_msg to peer [[15019,0],1] at tag 30
[cn2:00817] [[15019,1],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00817] [[15019,1],2] oob:base:send to target [[15019,0],1]
[cn2:00817] [[15019,1],2] oob:base:send known transport for peer [[15019,0],1]
[cn2:00817] [[15019,1],2] oob:tcp:send_nb to peer [[15019,0],1]:30
[cn2:00817] [[15019,1],2] tcp:send_nb to peer [[15019,0],1]
[cn2:00817] [[15019,1],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],1]
[cn2:00817] [[15019,1],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],1]:30
[cn2:00817] [[15019,1],2] routed_binomial_get([[15019,0],1]) --> [[15019,0],1]
[cn2:00817] [[15019,1],2] tcp:send_nb: already connected to [[15019,0],1] - queueing for send
[cn2:00817] [[15019,1],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],1]
[cn2:00817] [[15019,1],2] tcp:send_handler called to send to peer [[15019,0],1]
[cn2:00817] [[15019,1],2] tcp:send_handler SENDING TO [[15019,0],1]
[cn2:00817] [[15019,1],2] MESSAGE SEND COMPLETE TO [[15019,0],1] OF 348 BYTES ON SOCKET 11
[cn2:00817] [[15019,1],2]-[[15019,0],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 273 bytes from [[15019,0],1] for tag 2
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] message received 78 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 192 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 192
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 192 BYTES FOR DEST [[15019,0],0] TAG 2
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
2:00818] [[15019,1],3] MESSAGE SEND COMPLETE TO [[15019,0],1] OF 348 BYTES ON SOCKET 11
[cn2:00818] [[15019,1],3]-[[15019,0],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17680] usNIC connectivity client finalized
[cn1:14018] [[15019,0],0] message received 192 bytes from [[15019,0],3] for tag 2
[cn4:17680] mca: base: close: component usnic closed
[cn4:17680] mca: base: close: unloading component usnic
[cn4:17680] select: initializing btl component vader
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 668
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 668 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 180 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 668 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 668 bytes from [[15019,0],3] for tag 2
[cn4:17680] storing data in PEER dstore
[cn4:17680] dstore:hash:store storing data for proc 26754088961
[cn4:17680] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 26754088961
[cn4:17680] select: init of component vader returned success
[cn4:17680] select: init returned priority 20
[cn4:17680] selected ob1 best priority 20
[cn4:17680] select: component ob1 selected
[cn4:17680] mca: base: close: component v closed
[cn4:17680] mca: base: close: unloading component v
[cn4:17680] mca: base: close: component bfo closed
[cn4:17680] mca: base: close: unloading component bfo
[cn4:17680] [[15019,1],6] running modex
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 180
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 180 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 180 bytes from [[15019,0],3] for tag 2
[cn4:17680] [[15019,1],6] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1280
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1280 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 1280 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1280 BYTES ON SOCKET 9
[cn4:17681] btl:usnic: no usNICs found
[cn4:17681] storing data in PEER dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 31049056257
[cn4:17681] select: init of component usnic returned failure
[cn4:17681] mca: base: close: component usnic closed
[cn4:17681] mca: base: close: unloading component usnic
[cn4:17681] select: initializing btl component vader
[cn4:17681] storing data in PEER dstore
[cn4:17681] dstore:hash:store storing data for proc 31049056257
[cn4:17681] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 31049056257
[cn4:17681] select: init of component vader returned success
[cn4:17681] select: init returned priority 20
[cn4:17681] selected ob1 best priority 20
[cn4:17681] select: component ob1 selected
[cn4:17681] mca: base: close: component v closed
[cn4:17681] mca: base: close: unloading component v
[cn4:17681] mca: base: close: component bfo closed
[cn4:17681] mca: base: close: unloading component bfo
[cn4:17681] [[15019,1],7] running modex
[cn4:17681] [[15019,1],7] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3]:tcp:recv:handler called for peer [[15019,1],7]
[cn4:17677] [[15019,0],3]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 3119
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 3119 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3]:tcp:recv:handler allocate data region of size 348
[cn1:14018] [[15019,0],0] message received 3119 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17681] [[15019,1],7] grpcomm:base:modex: performing modex
[cn4:17681] fetching data from PEER dstore
[cn4:17681] dstore:hash:fetch: searching for key NULL on proc 31049056257
[cn4:17681] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 31049056257
[cn4:17681] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 31049056257
[cn4:17681] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 31049056257
[cn4:17681] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 31049056257
[cn4:17681] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 31049056257
[cn4:17681] [[15019,1],7] grpcomm:base:pack_modex: reporting 5 entries
[cn4:17681] [[15019,1],7] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn4:17681] [[15019,1],7] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn4:17681] [[15019,1],7] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn4:17681] [[15019,1],7] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn4:17681] [[15019,1],7] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn4:17681] [[15019,1],7] grpcomm:base:full:modex: executing allgather
[cn4:17681] [[15019,1],7] grpcomm:bad entering allgather
[cn4:17681] [[15019,1],7] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn4:17681] [[15019,1],7] grpcomm:base:modex: modex posted
[cn4:17681] [[15019,1],7] grpcomm:bad sending collective 0 to our daemon
[cn4:17681] [[15019,1],7] rml_send_buffer to peer [[15019,0],3] at tag 30
[cn4:17681] [[15019,1],7] grpcomm:bad allgather underway
[cn4:17681] [[15019,1],7] rml_send_msg to peer [[15019,0],3] at tag 30
[cn4:17681] [[15019,1],7] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17681] [[15019,1],7] oob:base:send to target [[15019,0],3]
[cn4:17681] [[15019,1],7] oob:base:send known transport for peer [[15019,0],3]
[cn4:17681] [[15019,1],7] oob:tcp:send_nb to peer [[15019,0],3]:30
[cn4:17681] [[15019,1],7] tcp:send_nb to peer [[15019,0],3]
[cn4:17681] [[15019,1],7]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],3]
[cn4:17681] [[15019,1],7]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],3]:30
[cn4:17681] [[15019,1],7] routed_binomial_get([[15019,0],3]) --> [[15019,0],3]
[cn4:17681] [[15019,1],7] tcp:send_nb: already connected to [[15019,0],3] - queueing for send
[cn4:17681] [[15019,1],7]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],3]
[cn4:17681] [[15019,1],7] tcp:send_handler called to send to peer [[15019,0],3]
[cn4:17681] [[15019,1],7] tcp:send_handler SENDING TO [[15019,0],3]
[cn4:17681] [[15019,1],7] MESSAGE SEND COMPLETE TO [[15019,0],3] OF 348 BYTES ON SOCKET 11
[cn4:17681] [[15019,1],7]-[[15019,0],3] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 3119 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 93
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 93 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 93 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 93 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17680] [[15019,1],6] grpcomm:base:modex: performing modex
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 2066
[cn4:17677] [[15019,0],3]:tcp:recv:handler called for peer [[15019,1],6]
[cn4:17677] [[15019,0],3]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 2066 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 2066 bytes from [[15019,0],3] for tag 2
[cn4:17680] fetching data from PEER dstore
[cn4:17680] dstore:hash:fetch: searching for key NULL on proc 26754088961
[cn4:17680] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 26754088961
[cn4:17680] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 26754088961
[cn4:17680] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 26754088961
[cn4:17680] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 26754088961
[cn4:17680] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 26754088961
[cn4:17680] [[15019,1],6] grpcomm:base:pack_modex: reporting 5 entries
[cn4:17680] [[15019,1],6] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn4:17680] [[15019,1],6] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn4:17680] [[15019,1],6] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn4:17680] [[15019,1],6] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn4:17680] [[15019,1],6] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn4:17680] [[15019,1],6] grpcomm:base:full:modex: executing allgather
[cn4:17680] [[15019,1],6] grpcomm:bad entering allgather
[cn4:17680] [[15019,1],6] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn4:17680] [[15019,1],6] grpcomm:base:modex: modex posted
[cn4:17680] [[15019,1],6] grpcomm:bad sending collective 0 to our daemon
[cn4:17680] [[15019,1],6] rml_send_buffer to peer [[15019,0],3] at tag 30
[cn4:17680] [[15019,1],6] grpcomm:bad allgather underway
[cn4:17680] [[15019,1],6] rml_send_msg to peer [[15019,0],3] at tag 30
[cn4:17680] [[15019,1],6] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17680] [[15019,1],6] oob:base:send to target [[15019,0],3]
[cn4:17680] [[15019,1],6] oob:base:send known transport for peer [[15019,0],3]
[cn4:17680] [[15019,1],6] oob:tcp:send_nb to peer [[15019,0],3]:30
[cn4:17680] [[15019,1],6] tcp:send_nb to peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],3]
[cn4:17677] [[15019,0],3]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn4:17677] [[15019,0],3]:tcp:recv:handler allocate data region of size 348
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],0] at tag 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1020
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 2066 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 1020 BYTES FOR DEST [[15019,0],0] TAG 2
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] message received 1020 bytes from [[15019,0],3] for tag 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],0] at tag 2
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],0]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],0]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn4:17680] [[15019,1],6]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],3]
[cn4:17680] [[15019,1],6]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],3]:30
[cn4:17680] [[15019,1],6] routed_binomial_get([[15019,0],3]) --> [[15019,0],3]
[cn4:17680] [[15019,1],6] tcp:send_nb: already connected to [[15019,0],3] - queueing for send
[cn4:17680] [[15019,1],6]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],3]
[cn4:17680] [[15019,1],6] tcp:send_handler called to send to peer [[15019,0],3]
[cn4:17680] [[15019,1],6] tcp:send_handler SENDING TO [[15019,0],3]
[cn4:17680] [[15019,1],6] MESSAGE SEND COMPLETE TO [[15019,0],3] OF 348 BYTES ON SOCKET 11
[cn4:17680] [[15019,1],6]-[[15019,0],3] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:2
[cn4:17677] [[15019,0],3] routing direct to the HNP
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],0]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],0]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1020 BYTES ON SOCKET 9
[cn4:17677] [[15019,0],3]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14033] [[15019,1],0]:tcp:recv:handler called for peer [[15019,0],0]
[cn1:14033] [[15019,1],0]:tcp:recv:handler CONNECTED
[cn1:14033] [[15019,1],0] RECVD COMPLETE MESSAGE FROM [[15019,0],0] (ORIGIN [[15019,0],0]) OF 14140 BYTES FOR DEST [[15019,1],0] TAG 20
[cn1:14033] [[15019,1],0] DELIVERING TO RML
[cn1:14033] [[15019,1],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14033] [[15019,1],0] message received 14140 bytes from [[15019,0],0] for tag 20
[cn1:14033] [[15019,1],0] registering sync ack recvd
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 984285184
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285184
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14033] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285185
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 984285185
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 5279252480
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252480
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14033] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 9574219776
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219776
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14033] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 13869187072
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187072
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14033] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 18164154368
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154368
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14033] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 18446744070398869505
[cn1:14033] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 18446744070398869505
[cn1:14033] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 984285185
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 984285185
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 984285185
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 984285185
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 5279252481
[cn1:14033] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5279252481
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 5279252481
[cn1:14033] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5279252481
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 5279252481
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 5279252481
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 5279252481
[cn1:14033] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5279252481
[cn1:14033] fetching data from INTERNAL dstore
[cn1:14033] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 5279252481
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252481
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 5279252481
[cn1:14033] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5279252481
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 9574219777
[cn1:14033] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9574219777
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 9574219777
[cn1:14033] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9574219777
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 9574219777
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 9574219777
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 9574219777
[cn1:14033] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9574219777
[cn1:14033] fetching data from INTERNAL dstore
[cn1:14033] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 9574219777
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219777
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 9574219777
[cn1:14033] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9574219777
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 13869187073
[cn1:14033] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13869187073
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 13869187073
[cn1:14033] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13869187073
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 13869187073
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 13869187073
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 13869187073
[cn1:14033] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13869187073
[cn1:14033] fetching data from INTERNAL dstore
[cn1:14033] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 13869187073
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187073
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 13869187073
[cn1:14033] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13869187073
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 18164154369
[cn1:14033] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18164154369
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 18164154369
[cn1:14033] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18164154369
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 18164154369
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 18164154369
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 18164154369
[cn1:14033] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18164154369
[cn1:14033] fetching data from INTERNAL dstore
[cn1:14033] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 18164154369
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154369
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 18164154369
[cn1:14033] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18164154369
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 22459121665
[cn1:14033] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22459121665
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 22459121665
[cn1:14033] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22459121665
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 22459121665
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 22459121665
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 22459121665
[cn1:14033] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22459121665
[cn1:14033] fetching data from INTERNAL dstore
[cn1:14033] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 22459121665
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22459121665
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 22459121665
[cn1:14033] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22459121665
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 26754088961
[cn1:14033] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26754088961
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 26754088961
[cn1:14033] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26754088961
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 26754088961
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 26754088961
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 26754088961
[cn1:14033] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26754088961
[cn1:14033] fetching data from INTERNAL dstore
[cn1:14033] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 26754088961
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26754088961
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 26754088961
[cn1:14033] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26754088961
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 31049056257
[cn1:14033] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31049056257
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 31049056257
[cn1:14033] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31049056257
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 31049056257
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 31049056257
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 31049056257
[cn1:14033] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31049056257
[cn1:14033] fetching data from INTERNAL dstore
[cn1:14033] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 31049056257
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31049056257
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 31049056257
[cn1:14033] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31049056257
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 35344023553
[cn1:14033] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35344023553
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 35344023553
[cn1:14033] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35344023553
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 35344023553
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 35344023553
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 35344023553
[cn1:14033] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35344023553
[cn1:14033] fetching data from INTERNAL dstore
[cn1:14033] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 35344023553
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35344023553
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 35344023553
[cn1:14033] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35344023553
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 39638990849
[cn1:14033] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39638990849
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 39638990849
[cn1:14033] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39638990849
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 39638990849
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 39638990849
[cn1:14033] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 39638990849
[cn1:14033] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39638990849
[cn1:14033] fetching data from INTERNAL dstore
[cn1:14033] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 39638990849
[cn1:14033] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39638990849
[cn1:14033] storing data in INTERNAL dstore
[cn1:14033] dstore:hash:store storing data for proc 39638990849
[cn1:14033] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39638990849
[cn1:14033] [[15019,1],0] Process bound at launch
[cn1:14033] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B/././.][./././.]
[cn1:14033] storing data in NONPEER dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 984285185
[cn1:14033] storing data in PEER dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 984285185
[cn1:14034] [[15019,1],1]:tcp:recv:handler called for peer [[15019,0],0]
[cn1:14034] [[15019,1],1]:tcp:recv:handler CONNECTED
[cn1:14034] [[15019,1],1] RECVD COMPLETE MESSAGE FROM [[15019,0],0] (ORIGIN [[15019,0],0]) OF 14140 BYTES FOR DEST [[15019,1],1] TAG 20
[cn1:14034] [[15019,1],1] DELIVERING TO RML
[cn1:14034] [[15019,1],1] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14034] [[15019,1],1] message received 14140 bytes from [[15019,0],0] for tag 20
[cn1:14034] [[15019,1],1] registering sync ack recvd
[cn1:14033] mca: base: components_register: registering bml components
[cn1:14033] mca: base: components_register: found loaded component r2
[cn1:14033] mca: base: components_register: component r2 register function successful
[cn1:14033] mca: base: components_open: opening bml components
[cn1:14033] mca: base: components_open: found loaded component r2
[cn1:14033] mca: base: components_open: component r2 open function successful
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 984285184
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285184
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14034] dstore:hash:store: storing key cn1[OPAL_UINT32] for proc 18446744069414584318
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252481
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 5279252481
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252480
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 5279252480
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14034] dstore:hash:store: storing key cn2[OPAL_UINT32] for proc 18446744069414584318
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 9574219776
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219776
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14034] dstore:hash:store: storing key cn3[OPAL_UINT32] for proc 18446744069414584318
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 13869187072
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187072
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14034] dstore:hash:store: storing key cn4[OPAL_UINT32] for proc 18446744069414584318
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 18164154368
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154368
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 18446744069414584318
[cn1:14034] dstore:hash:store: storing key cn5[OPAL_UINT32] for proc 18446744069414584318
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 18446744070398869505
[cn1:14034] dstore:hash:store: storing key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 18446744070398869505
[cn1:14034] dstore:hash:store: updating key orte.nprocs[OPAL_UINT32] for proc 18446744070398869505
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key opal.local.ldr[OPAL_UINT64] for proc 5279252481
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 984285185
[cn1:14034] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 984285185
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 984285185
[cn1:14034] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 984285185
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 984285185
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 984285185
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 984285185
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 984285185
[cn1:14034] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 984285185
[cn1:14034] fetching data from INTERNAL dstore
[cn1:14034] dstore:hash:fetch: searching for key orte.hostname on proc 984285184
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 984285185
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 984285185
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 984285185
[cn1:14034] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 984285185
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 5279252481
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 5279252481
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 5279252481
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 9574219777
[cn1:14034] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 9574219777
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 9574219777
[cn1:14034] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 9574219777
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 9574219777
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 9574219777
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 9574219777
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 9574219777
[cn1:14034] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 9574219777
[cn1:14034] fetching data from INTERNAL dstore
[cn1:14034] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 9574219777
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 9574219777
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 9574219777
[cn1:14034] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 9574219777
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 13869187073
[cn1:14034] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 13869187073
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 13869187073
[cn1:14034] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 13869187073
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 13869187073
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 13869187073
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 13869187073
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 13869187073
[cn1:14034] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 13869187073
[cn1:14034] fetching data from INTERNAL dstore
[cn1:14034] dstore:hash:fetch: searching for key orte.hostname on proc 5279252480
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 13869187073
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 13869187073
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 13869187073
[cn1:14034] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 13869187073
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 18164154369
[cn1:14034] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 18164154369
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 18164154369
[cn1:14034] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 18164154369
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 18164154369
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 18164154369
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 18164154369
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 18164154369
[cn1:14034] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 18164154369
[cn1:14034] fetching data from INTERNAL dstore
[cn1:14034] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 18164154369
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 18164154369
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 18164154369
[cn1:14034] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 18164154369
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 22459121665
[cn1:14034] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 22459121665
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 22459121665
[cn1:14034] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 22459121665
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 22459121665
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 22459121665
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 22459121665
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 22459121665
[cn1:14034] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 22459121665
[cn1:14034] fetching data from INTERNAL dstore
[cn1:14034] dstore:hash:fetch: searching for key orte.hostname on proc 9574219776
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 22459121665
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 22459121665
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 22459121665
[cn1:14034] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 22459121665
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 26754088961
[cn1:14034] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 26754088961
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 26754088961
[cn1:14034] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 26754088961
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 26754088961
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 26754088961
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 26754088961
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 26754088961
[cn1:14034] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 26754088961
[cn1:14034] fetching data from INTERNAL dstore
[cn1:14034] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 26754088961
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 26754088961
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 26754088961
[cn1:14034] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 26754088961
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 31049056257
[cn1:14034] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 31049056257
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 31049056257
[cn1:14034] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 31049056257
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 31049056257
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 31049056257
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 31049056257
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 31049056257
[cn1:14034] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 31049056257
[cn1:14034] fetching data from INTERNAL dstore
[cn1:14034] dstore:hash:fetch: searching for key orte.hostname on proc 13869187072
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 31049056257
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 31049056257
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 31049056257
[cn1:14034] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 31049056257
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 35344023553
[cn1:14034] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 35344023553
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 35344023553
[cn1:14034] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 35344023553
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 35344023553
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 35344023553
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 35344023553
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 35344023553
[cn1:14034] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 35344023553
[cn1:14034] fetching data from INTERNAL dstore
[cn1:14034] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 35344023553
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 35344023553
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 35344023553
[cn1:14034] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 35344023553
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 39638990849
[cn1:14034] dstore:hash:store: storing key opal.local.rank[OPAL_UINT16] for proc 39638990849
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 39638990849
[cn1:14034] dstore:hash:store: storing key orte.node.rank[OPAL_UINT16] for proc 39638990849
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 39638990849
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 39638990849
[cn1:14034] dstore:hash:store: storing key opal.cpuset[OPAL_STRING] for proc 39638990849
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 39638990849
[cn1:14034] dstore:hash:store: storing key orte.daemon.vpid[OPAL_UINT32] for proc 39638990849
[cn1:14034] fetching data from INTERNAL dstore
[cn1:14034] dstore:hash:fetch: searching for key orte.hostname on proc 18164154368
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 39638990849
[cn1:14034] dstore:hash:store: storing key orte.hostname[OPAL_STRING] for proc 39638990849
[cn1:14034] storing data in INTERNAL dstore
[cn1:14034] dstore:hash:store storing data for proc 39638990849
[cn1:14034] dstore:hash:store: storing key orte.global.rank[OPAL_UINT32] for proc 39638990849
[cn1:14034] [[15019,1],1] Process bound at launch
[cn1:14034] MCW rank 1 bound to socket 0[core 1[hwt 0]]: [./B/./.][./././.]
[cn1:14034] storing data in NONPEER dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: updating key opal.cpuset[OPAL_STRING] for proc 5279252481
[cn1:14034] storing data in PEER dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key MPI_THREAD_LEVEL[OPAL_BYTE_OBJECT] for proc 5279252481
[cn1:14034] mca: base: components_register: registering bml components
[cn1:14034] mca: base: components_register: found loaded component r2
[cn1:14034] mca: base: components_register: component r2 register function successful
[cn1:14034] mca: base: components_open: opening bml components
[cn1:14034] mca: base: components_open: found loaded component r2
[cn1:14034] mca: base: components_open: component r2 open function successful
[cn1:14033] mca: base: components_register: registering btl components
[cn1:14033] mca: base: components_register: found loaded component openib
[cn1:14033] mca: base: components_register: component openib register function successful
[cn1:14033] mca: base: components_register: found loaded component self
[cn1:14033] mca: base: components_register: component self register function successful
[cn1:14033] mca: base: components_register: found loaded component sm
[cn1:14033] mca: base: components_register: component sm register function successful
[cn1:14033] mca: base: components_register: found loaded component tcp
[cn1:14033] mca: base: components_register: component tcp register function successful
[cn1:14033] mca: base: components_register: found loaded component usnic
[cn1:14033] mca: base: components_register: component usnic register function successful
[cn1:14033] mca: base: components_register: found loaded component vader
[cn1:14033] mca: base: components_register: component vader register function successful
[cn1:14033] mca: base: components_open: opening btl components
[cn1:14033] mca: base: components_open: found loaded component openib
[cn1:14033] mca: base: components_open: component openib open function successful
[cn1:14033] mca: base: components_open: found loaded component self
[cn1:14033] mca: base: components_open: component self open function successful
[cn1:14033] mca: base: components_open: found loaded component sm
[cn1:14033] mca: base: components_open: component sm open function successful
[cn1:14033] mca: base: components_open: found loaded component tcp
[cn1:14033] mca: base: components_open: component tcp open function successful
[cn1:14033] mca: base: components_open: found loaded component usnic
[cn1:14033] mca: base: components_open: component usnic open function successful
[cn1:14033] mca: base: components_open: found loaded component vader
[cn1:14033] mca: base: components_open: component vader open function successful
[cn1:14034] mca: base: components_register: registering btl components
[cn1:14034] mca: base: components_register: found loaded component openib
[cn1:14034] mca: base: components_register: component openib register function successful
[cn1:14034] mca: base: components_register: found loaded component self
[cn5:14014] [[15019,0],4]:tcp:recv:handler called for peer [[15019,1],8]
[cn5:14014] [[15019,0],4]:tcp:recv:handler CONNECTED
[cn1:14033] mca: base: components_register: registering pml components
[cn1:14033] mca: base: components_register: found loaded component v
[cn1:14033] mca: base: components_register: component v register function successful
[cn1:14033] mca: base: components_register: found loaded component bfo
[cn1:14034] mca: base: components_register: component self register function successful
[cn1:14034] mca: base: components_register: found loaded component sm
[cn1:14034] mca: base: components_register: component sm register function successful
[cn1:14034] mca: base: components_register: found loaded component tcp
[cn1:14034] mca: base: components_register: component tcp register function successful
[cn1:14034] mca: base: components_register: found loaded component usnic
[cn1:14034] mca: base: components_register: component usnic register function successful
[cn1:14034] mca: base: components_register: found loaded component vader
[cn1:14034] mca: base: components_register: component vader register function successful
[cn1:14034] mca: base: components_open: opening btl components
[cn1:14034] mca: base: components_open: found loaded component openib
[cn1:14034] mca: base: components_open: component openib open function successful
[cn1:14034] mca: base: components_open: found loaded component self
[cn1:14034] mca: base: components_open: component self open function successful
[cn1:14034] mca: base: components_open: found loaded component sm
[cn1:14034] mca: base: components_open: component sm open function successful
[cn1:14034] mca: base: components_open: found loaded component tcp
[cn1:14034] mca: base: components_open: component tcp open function successful
[cn1:14034] mca: base: components_open: found loaded component usnic
[cn1:14034] mca: base: components_open: component usnic open function successful
[cn1:14034] mca: base: components_open: found loaded component vader
[cn1:14034] mca: base: components_open: component vader open function successful
[cn5:14014] [[15019,0],4] RECVD COMPLETE MESSAGE FROM [[15019,1],8] (ORIGIN [[15019,1],8]) OF 348 BYTES FOR DEST [[15019,0],4] TAG 30
[cn5:14014] [[15019,0],4] DELIVERING TO RML
[cn5:14014] [[15019,0],4] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn5:14014] [[15019,0],4] message received 348 bytes from [[15019,1],8] for tag 30
[cn5:14014] [[15019,0],4] COLLECTIVE RECVD FROM [[15019,1],8]
[cn5:14014] [[15019,0],4] WORKING COLLECTIVE 0
[cn5:14014] [[15019,0],4] PROGRESSING COLLECTIVE 0
[cn5:14014] [[15019,0],4] PROGRESSING COLL id 0
[cn5:14014] [[15019,0],4] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn5:14014] [[15019,0],4] PROGRESSING COLL id 1
[cn5:14014] [[15019,0],4] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn5:14014] [[15019,0],4] PROGRESSING COLL id 2
[cn5:14014] [[15019,0],4] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn1:14033] mca: base: components_register: component bfo register function successful
[cn1:14033] mca: base: components_register: found loaded component cm
[cn1:14033] mca: base: components_register: component cm register function successful
[cn1:14033] mca: base: components_register: found loaded component ob1
[cn1:14033] mca: base: components_register: component ob1 register function successful
[cn1:14033] mca: base: components_open: opening pml components
[cn1:14033] mca: base: components_open: found loaded component v
[cn1:14033] mca: base: components_open: component v open function successful
[cn1:14033] mca: base: components_open: found loaded component bfo
[cn1:14033] mca: base: components_open: component bfo open function successful
[cn1:14033] mca: base: components_open: found loaded component cm
[cn1:14034] mca: base: components_register: registering pml components
[cn1:14034] mca: base: components_register: found loaded component v
[cn1:14034] mca: base: components_register: component v register function successful
[cn1:14034] mca: base: components_register: found loaded component bfo
[cn1:14034] mca: base: components_register: component bfo register function successful
[cn1:14034] mca: base: components_register: found loaded component cm
[cn1:14034] mca: base: components_register: component cm register function successful
[cn1:14034] mca: base: components_register: found loaded component ob1
[cn1:14034] mca: base: components_register: component ob1 register function successful
[cn1:14034] mca: base: components_open: opening pml components
[cn1:14034] mca: base: components_open: found loaded component v
[cn1:14034] mca: base: components_open: component v open function successful
[cn1:14034] mca: base: components_open: found loaded component bfo
[cn1:14034] mca: base: components_open: component bfo open function successful
[cn1:14034] mca: base: components_open: found loaded component cm
[cn3:21265] [[15019,0],2]:tcp:recv:handler called for peer [[15019,1],5]
[cn3:21265] [[15019,0],2]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] RECVD COMPLETE MESSAGE FROM [[15019,1],5] (ORIGIN [[15019,1],5]) OF 348 BYTES FOR DEST [[15019,0],2] TAG 30
[cn3:21265] [[15019,0],2] DELIVERING TO RML
[cn3:21265] [[15019,0],2] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] message received 348 bytes from [[15019,1],5] for tag 30
[cn3:21265] [[15019,0],2] COLLECTIVE RECVD FROM [[15019,1],5]
[cn3:21265] [[15019,0],2] WORKING COLLECTIVE 0
[cn3:21265] [[15019,0],2] PROGRESSING COLLECTIVE 0
[cn3:21265] [[15019,0],2] PROGRESSING COLL id 0
[cn3:21265] [[15019,0],2] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn3:21265] [[15019,0],2] PROGRESSING COLL id 1
[cn3:21265] [[15019,0],2] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn3:21265] [[15019,0],2] PROGRESSING COLL id 2
[cn3:21265] [[15019,0],2] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn2:00814] [[15019,0],1]:tcp:recv:handler called for peer [[15019,1],3]
[cn2:00814] [[15019,0],1]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] RECVD COMPLETE MESSAGE FROM [[15019,1],3] (ORIGIN [[15019,1],3]) OF 348 BYTES FOR DEST [[15019,0],1] TAG 30
[cn2:00814] [[15019,0],1] DELIVERING TO RML
[cn2:00814] [[15019,0],1] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] message received 348 bytes from [[15019,1],3] for tag 30
[cn2:00814] [[15019,0],1] COLLECTIVE RECVD FROM [[15019,1],3]
[cn2:00814] [[15019,0],1] WORKING COLLECTIVE 0
[cn2:00814] [[15019,0],1] PROGRESSING COLLECTIVE 0
[cn2:00814] [[15019,0],1] PROGRESSING COLL id 0
[cn2:00814] [[15019,0],1] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn2:00814] [[15019,0],1] PROGRESSING COLL id 1
[cn2:00814] [[15019,0],1] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn2:00814] [[15019,0],1] PROGRESSING COLL id 2
[cn2:00814] [[15019,0],1] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn3:21265] [[15019,0],2]:tcp:recv:handler called for peer [[15019,1],4]
[cn3:21265] [[15019,0],2]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2] RECVD COMPLETE MESSAGE FROM [[15019,1],4] (ORIGIN [[15019,1],4]) OF 348 BYTES FOR DEST [[15019,0],2] TAG 30
[cn3:21265] [[15019,0],2] DELIVERING TO RML
[cn3:21265] [[15019,0],2] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn3:21265] [[15019,0],2] message received 348 bytes from [[15019,1],4] for tag 30
[cn3:21265] [[15019,0],2] COLLECTIVE RECVD FROM [[15019,1],4]
[cn3:21265] [[15019,0],2] WORKING COLLECTIVE 0
[cn3:21265] [[15019,0],2] PROGRESSING COLLECTIVE 0
[cn3:21265] [[15019,0],2] PROGRESSING COLL id 0
[cn3:21265] [[15019,0],2] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn3:21265] [[15019,0],2] COLLECTIVE 0 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],2] at tag 32
[cn3:21265] [[15019,0],2] PROGRESSING COLL id 1
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 708
[cn3:21265] [[15019,0],2] PROGRESSING COLL id 2
[cn3:21265] [[15019,0],2] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],2] at tag 32
[cn3:21265] [[15019,0],2] rml_send_iovec_to_self at tag 32
[cn3:21265] [[15019,0],2] rml_send_to_self callback executing for tag 32
[cn3:21265] [[15019,0],2] message received 708 bytes from [[15019,0],2] for tag 32
[cn3:21265] [[15019,0],2] grpcomm:base:daemon_coll: daemon collective recvd from [[15019,0],2]
[cn3:21265] [[15019,0],2] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn3:21265] [[15019,0],2] grpcomm:base:daemon_coll: NUM CONTRIBS: 2
[cn3:21265] [[15019,0],2] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[15019,0],0]
[cn3:21265] [[15019,0],2] rml_send_buffer to peer [[15019,0],0] at tag 32
[cn3:21265] [[15019,0],2] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 10 ngr 2
[cn3:21265] [[15019,0],2] rml_send_msg to peer [[15019,0],0] at tag 32
[cn3:21265] [[15019,0],2] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn3:21265] [[15019,0],2] oob:base:send to target [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:base:send known transport for peer [[15019,0],0]
[cn3:21265] [[15019,0],2] oob:tcp:send_nb to peer [[15019,0],0]:32
[cn3:21265] [[15019,0],2] tcp:send_nb to peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:32
[cn3:21265] [[15019,0],2] routing direct to the HNP
[cn3:21265] [[15019,0],2] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn3:21265] [[15019,0],2]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler called to send to peer [[15019,0],0]
[cn3:21265] [[15019,0],2] tcp:send_handler SENDING TO [[15019,0],0]
[cn3:21265] [[15019,0],2] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 708 BYTES ON SOCKET 9
[cn3:21265] [[15019,0],2]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],2] (ORIGIN [[15019,0],2]) OF 708 BYTES FOR DEST [[15019,0],0] TAG 32
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 708 bytes from [[15019,0],2] for tag 32
[cn1:14018] [[15019,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[15019,0],2]
[cn1:14018] [[15019,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn1:14018] [[15019,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 2
[cn1:14018] [[15019,0],0] grpcomm:base:daemon_coll: CANNOT PROGRESS
[cn2:00814] [[15019,0],1]:tcp:recv:handler called for peer [[15019,1],2]
[cn2:00814] [[15019,0],1]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] RECVD COMPLETE MESSAGE FROM [[15019,1],2] (ORIGIN [[15019,1],2]) OF 348 BYTES FOR DEST [[15019,0],1] TAG 30
[cn2:00814] [[15019,0],1] DELIVERING TO RML
[cn2:00814] [[15019,0],1] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] message received 348 bytes from [[15019,1],2] for tag 30
[cn2:00814] [[15019,0],1] COLLECTIVE RECVD FROM [[15019,1],2]
[cn2:00814] [[15019,0],1] WORKING COLLECTIVE 0
[cn2:00814] [[15019,0],1] PROGRESSING COLLECTIVE 0
[cn2:00814] [[15019,0],1] PROGRESSING COLL id 0
[cn2:00814] [[15019,0],1] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn2:00814] [[15019,0],1] COLLECTIVE 0 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],1] at tag 32
[cn2:00814] [[15019,0],1] PROGRESSING COLL id 1
[cn4:17677] [[15019,0],3]:tcp:recv:handler called for peer [[15019,1],7]
[cn4:17677] [[15019,0],3]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] RECVD COMPLETE MESSAGE FROM [[15019,1],7] (ORIGIN [[15019,1],7]) OF 348 BYTES FOR DEST [[15019,0],3] TAG 30
[cn4:17677] [[15019,0],3] DELIVERING TO RML
[cn4:17677] [[15019,0],3] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] message received 348 bytes from [[15019,1],7] for tag 30
[cn4:17677] [[15019,0],3] COLLECTIVE RECVD FROM [[15019,1],7]
[cn4:17677] [[15019,0],3] WORKING COLLECTIVE 0
[cn4:17677] [[15019,0],3] PROGRESSING COLLECTIVE 0
[cn4:17677] [[15019,0],3] PROGRESSING COLL id 0
[cn4:17677] [[15019,0],3] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn4:17677] [[15019,0],3] PROGRESSING COLL id 1
[cn4:17677] [[15019,0],3] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn4:17677] [[15019,0],3] PROGRESSING COLL id 2
[cn4:17677] [[15019,0],3] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn2:00814] [[15019,0],1] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn2:00814] [[15019,0],1] PROGRESSING COLL id 2
[cn2:00814] [[15019,0],1] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],1] at tag 32
[cn2:00814] [[15019,0],1] rml_send_iovec_to_self at tag 32
[cn2:00814] [[15019,0],1] rml_send_to_self callback executing for tag 32
[cn2:00814] [[15019,0],1] message received 708 bytes from [[15019,0],1] for tag 32
[cn2:00814] [[15019,0],1] grpcomm:base:daemon_coll: daemon collective recvd from [[15019,0],1]
[cn2:00814] [[15019,0],1] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn2:00814] [[15019,0],1] grpcomm:base:daemon_coll: NUM CONTRIBS: 2
[cn2:00814] [[15019,0],1] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 10 ngr 2
[cn4:17677] [[15019,0],3]:tcp:recv:handler called for peer [[15019,1],6]
[cn4:17677] [[15019,0],3]:tcp:recv:handler CONNECTED
[cn4:17677] [[15019,0],3] RECVD COMPLETE MESSAGE FROM [[15019,1],6] (ORIGIN [[15019,1],6]) OF 348 BYTES FOR DEST [[15019,0],3] TAG 30
[cn4:17677] [[15019,0],3] DELIVERING TO RML
[cn4:17677] [[15019,0],3] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn4:17677] [[15019,0],3] message received 348 bytes from [[15019,1],6] for tag 30
[cn4:17677] [[15019,0],3] COLLECTIVE RECVD FROM [[15019,1],6]
[cn4:17677] [[15019,0],3] WORKING COLLECTIVE 0
[cn4:17677] [[15019,0],3] PROGRESSING COLLECTIVE 0
[cn4:17677] [[15019,0],3] PROGRESSING COLL id 0
[cn4:17677] [[15019,0],3] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn4:17677] [[15019,0],3] COLLECTIVE 0 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],3] at tag 32
[cn4:17677] [[15019,0],3] PROGRESSING COLL id 1
[cn4:17677] [[15019,0],3] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn4:17677] [[15019,0],3] PROGRESSING COLL id 2
[cn4:17677] [[15019,0],3] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn2:00814] [[15019,0],1]:tcp:recv:handler called for peer [[15019,0],3]
[cn2:00814] [[15019,0],1]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1]:tcp:recv:handler allocate data region of size 708
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],3] at tag 32
[cn4:17677] [[15019,0],3] rml_send_iovec_to_self at tag 32
[cn4:17677] [[15019,0],3] rml_send_to_self callback executing for tag 32
[cn4:17677] [[15019,0],3] message received 708 bytes from [[15019,0],3] for tag 32
[cn4:17677] [[15019,0],3] grpcomm:base:daemon_coll: daemon collective recvd from [[15019,0],3]
[cn4:17677] [[15019,0],3] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn4:17677] [[15019,0],3] grpcomm:base:daemon_coll: NUM CONTRIBS: 2
[cn4:17677] [[15019,0],3] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[15019,0],1]
[cn4:17677] [[15019,0],3] rml_send_buffer to peer [[15019,0],1] at tag 32
[cn4:17677] [[15019,0],3] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 10 ngr 2
[cn4:17677] [[15019,0],3] rml_send_msg to peer [[15019,0],1] at tag 32
[cn4:17677] [[15019,0],3] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn4:17677] [[15019,0],3] oob:base:send to target [[15019,0],1]
[cn4:17677] [[15019,0],3] oob:base:send known transport for peer [[15019,0],1]
[cn4:17677] [[15019,0],3] oob:tcp:send_nb to peer [[15019,0],1]:32
[cn4:17677] [[15019,0],3] tcp:send_nb to peer [[15019,0],1]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],1]
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],1]:32
[cn4:17677] [[15019,0],3] routed_binomial_get([[15019,0],1]) --> [[15019,0],1]
[cn4:17677] [[15019,0],3] tcp:send_nb: already connected to [[15019,0],1] - queueing for send
[cn4:17677] [[15019,0],3]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],1]
[cn4:17677] [[15019,0],3] tcp:send_handler called to send to peer [[15019,0],1]
[cn4:17677] [[15019,0],3] tcp:send_handler SENDING TO [[15019,0],1]
[cn4:17677] [[15019,0],3] MESSAGE SEND COMPLETE TO [[15019,0],1] OF 708 BYTES ON SOCKET 10
[cn4:17677] [[15019,0],3]-[[15019,0],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1]:tcp:recv:handler called for peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn2:00814] [[15019,0],1]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1] RECVD COMPLETE MESSAGE FROM [[15019,0],3] (ORIGIN [[15019,0],3]) OF 708 BYTES FOR DEST [[15019,0],1] TAG 32
[cn2:00814] [[15019,0],1] DELIVERING TO RML
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn2:00814] [[15019,0],1] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn2:00814] [[15019,0],1] message received 708 bytes from [[15019,0],3] for tag 32
[cn2:00814] [[15019,0],1] grpcomm:base:daemon_coll: daemon collective recvd from [[15019,0],3]
[cn2:00814] [[15019,0],1] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn2:00814] [[15019,0],1] grpcomm:base:daemon_coll: NUM CONTRIBS: 2
[cn2:00814] [[15019,0],1] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[15019,0],0]
[cn2:00814] [[15019,0],1] rml_send_buffer to peer [[15019,0],0] at tag 32
[cn2:00814] [[15019,0],1] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 10 ngr 4
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 1384
[cn2:00814] [[15019,0],1] rml_send_msg to peer [[15019,0],0] at tag 32
[cn2:00814] [[15019,0],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn2:00814] [[15019,0],1] oob:base:send to target [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:base:send known transport for peer [[15019,0],0]
[cn2:00814] [[15019,0],1] oob:tcp:send_nb to peer [[15019,0],0]:32
[cn2:00814] [[15019,0],1] tcp:send_nb to peer [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:32
[cn2:00814] [[15019,0],1] routing direct to the HNP
[cn2:00814] [[15019,0],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn2:00814] [[15019,0],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn2:00814] [[15019,0],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],1]
[cn2:00814] [[15019,0],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 1384 BYTES ON SOCKET 9
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn2:00814] [[15019,0],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,0],1] (ORIGIN [[15019,0],1]) OF 1384 BYTES FOR DEST [[15019,0],0] TAG 32
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 1384 bytes from [[15019,0],1] for tag 32
[cn1:14018] [[15019,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[15019,0],1]
[cn1:14018] [[15019,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn1:14018] [[15019,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 4
[cn1:14018] [[15019,0],0] grpcomm:base:daemon_coll: CANNOT PROGRESS
[cn1:14033] mca: base: components_register: registering mtl components
[cn1:14033] mca: base: components_register: found loaded component psm
[cn1:14033] mca: base: components_register: component psm register function successful
[cn1:14033] mca: base: components_open: opening mtl components
[cn1:14033] mca: base: components_open: found loaded component psm
[cn1:14033] mca: base: close: component psm closed
[cn1:14033] mca: base: close: unloading component psm
[cn1:14033] mca: base: close: component cm closed
[cn1:14033] mca: base: close: unloading component cm
[cn1:14033] mca: base: components_open: found loaded component ob1
[cn1:14033] mca: base: components_open: component ob1 open function successful
[cn1:14034] mca: base: components_register: registering mtl components
[cn1:14034] mca: base: components_register: found loaded component psm
[cn1:14034] mca: base: components_register: component psm register function successful
[cn1:14034] mca: base: components_open: opening mtl components
[cn1:14034] mca: base: components_open: found loaded component psm
[cn1:14034] mca: base: close: component psm closed
[cn1:14034] mca: base: close: unloading component psm
[cn1:14034] mca: base: close: component cm closed
[cn1:14034] mca: base: close: unloading component cm
[cn1:14034] mca: base: components_open: found loaded component ob1
[cn1:14034] mca: base: components_open: component ob1 open function successful
[cn1:14034] select: component v not in the include list
[cn1:14033] select: component v not in the include list
[cn1:14033] select: component bfo not in the include list
[cn1:14033] select: initializing pml component ob1
[cn1:14033] select: initializing btl component openib
[cn1:14034] select: component bfo not in the include list
[cn1:14034] select: initializing pml component ob1
[cn1:14034] select: initializing btl component openib
[cn1][[15019,1],0][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn1][[15019,1],1][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:686:init_one_port] looking for mlx4_0:1 GID index 0
[cn1][[15019,1],1][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn1][[15019,1],0][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:717:init_one_port] my IB subnet_id for HCA mlx4_0 port 1 is fe80000000000000
[cn1][[15019,1],0][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn1][[15019,1],0][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn1][[15019,1],0][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn1][[15019,1],0][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn1][[15019,1],0][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn1][[15019,1],0][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn1][[15019,1],0][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn1:14033] [rank=0] openib: using device mlx4_0
[cn1:14033] openib BTL: rdmacm IP address not found on port
[cn1:14033] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn1][[15019,1],1][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1294:setup_qps] pp: rd_num is 256 rd_low is 192 rd_win 128 rd_rsv 4
[cn1][[15019,1],1][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn1][[15019,1],1][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn1][[15019,1],1][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_component.c:1339:setup_qps] srq: rd_num is 1024 rd_low is 1008 sd_max is 64 rd_max is 256 srq_limit is 48
[cn1][[15019,1],1][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:2060:rdmacm_component_query] rdmacm_component_query
[cn1][[15019,1],1][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib_ip.c:132:mca_btl_openib_rdma_get_ipv4addr] Looking for mlx4_0:1 in IP address list
[cn1][[15019,1],1][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_rdmacm.c:1948:ipaddrcheck] *** Could not find IP address for mlx4_0:1 -- is there an IP address configured for this device?
[cn1][[15019,1],1][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x250a5a0 for btl 0x25180e0
[cn1:14034] [rank=1] openib: using device mlx4_0
[cn1:14034] openib BTL: rdmacm IP address not found on port
[cn1:14034] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cn1][[15019,1],0][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:543:udcm_module_init] created cpc module 0x15bd5a0 for btl 0x15cb0e0
[cn1][[15019,1],1][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] [cn1][[15019,1],0][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:793:udcm_module_create_listen_qp] creating listen QP on port 1
creating listen QP on port 1
[cn1][[15019,1],0][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 3, qpn 37173
[cn1][[15019,1],0][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 3, Port: 1, QPN: 37173
[cn1][[15019,1],0][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn1:14033] storing data in PEER dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 984285185
[cn1:14033] select: init of component openib returned success
[cn1:14033] select: initializing btl component self
[cn1:14033] select: init of component self returned success
[cn1:14033] select: initializing btl component sm
[cn1][[15019,1],1][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:854:udcm_module_create_listen_qp] listening for connections on lid 3, qpn 37174
[cn1][[15019,1],1][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:598:udcm_module_init] my modex = LID: 3, Port: 1, QPN: 37174[cn1:14033] select: init of component sm returned success
[cn1:14033] select: initializing btl component tcp
[cn1:14033] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn1:14033] btl: tcp: Found match: 127.0.0.1 (lo)
[cn1:14033] storing data in PEER dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 984285185
[cn1:14033] select: init of component tcp returned success
[cn1:14033] select: initializing btl component usnic
[cn1:14033] usNIC connectivity agent initialized
[cn1:14034] storing data in PEER dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key btl.openib.1.9[OPAL_BYTE_OBJECT] for proc 5279252481
[cn1:14034] select: init of component openib returned success
[cn1:14034] select: initializing btl component self
[cn1:14034] select: init of component self returned success
[cn1:14034] select: initializing btl component sm
[cn1:14034] select: init of component sm returned success
[cn1:14034] select: initializing btl component tcp
[cn1:14034] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[cn1:14034] btl: tcp: Found match: 127.0.0.1 (lo)
[cn1:14034] storing data in PEER dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key btl.tcp.1.9[OPAL_BYTE_OBJECT] for proc 5279252481
[cn1:14034] select: init of component tcp returned success
[cn1:14034] select: initializing btl component usnic

[cn1][[15019,1],1][/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/connect/btl_openib_connect_udcm.c:472:udcm_component_query] available for use on mlx4_0:1
[cn1:14034] usNIC connectivity client initialized
[cn1:14034] found 1 verbs interface
[cn1:14034] examining verbs interface: mlx4_0
[cn1:14033] usNIC connectivity client initialized
[cn1:14033] found 1 verbs interface
[cn1:14033] examining verbs interface: mlx4_0
[cn1:14034] found 1 verbs interface
[cn1:14034] examining verbs interface: mlx4_0
[cn1:14033] found 1 verbs interface
[cn1:14033] examining verbs interface: mlx4_0
[cn1:14033] btl:usnic: no usNICs found
[cn1:14033] storing data in PEER dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 984285185
[cn1:14033] select: init of component usnic returned failure
[cn1:14034] btl:usnic: no usNICs found
[cn1:14034] storing data in PEER dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key btl.usnic.1.9[OPAL_BYTE_OBJECT] for proc 5279252481
[cn1:14034] select: init of component usnic returned failure
[cn1:14034] mca: base: close: component usnic closed
[cn1:14034] mca: base: close: unloading component usnic
[cn1:14034] select: initializing btl component vader
[cn1:14033] usNIC connectivity client finalized
[cn1:14033] mca: base: close: component usnic closed
[cn1:14033] mca: base: close: unloading component usnic
[cn1:14033] select: initializing btl component vader
[cn1:14033] storing data in PEER dstore
[cn1:14033] dstore:hash:store storing data for proc 984285185
[cn1:14033] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 984285185
[cn1:14033] select: init of component vader returned success
[cn1:14033] select: init returned priority 20
[cn1:14033] selected ob1 best priority 20
[cn1:14033] select: component ob1 selected
[cn1:14033] mca: base: close: component v closed
[cn1:14033] mca: base: close: unloading component v
[cn1:14033] mca: base: close: component bfo closed
[cn1:14033] mca: base: close: unloading component bfo
[cn1:14033] [[15019,1],0] running modex
[cn1:14033] [[15019,1],0] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn1:14034] storing data in PEER dstore
[cn1:14034] dstore:hash:store storing data for proc 5279252481
[cn1:14034] dstore:hash:store: storing key btl.vader.1.9[OPAL_BYTE_OBJECT] for proc 5279252481
[cn1:14034] select: init of component vader returned success
[cn1:14034] select: init returned priority 20
[cn1:14034] selected ob1 best priority 20
[cn1:14034] select: component ob1 selected
[cn1:14034] mca: base: close: component v closed
[cn1:14034] mca: base: close: unloading component v
[cn1:14034] mca: base: close: component bfo closed
[cn1:14034] mca: base: close: unloading component bfo
[cn1:14034] [[15019,1],1] running modex
[cn1:14034] [[15019,1],1] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:325
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,1],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 372
[cn1:14034] [[15019,1],1] grpcomm:base:modex: performing modex
[cn1:14034] fetching data from PEER dstore
[cn1:14034] dstore:hash:fetch: searching for key NULL on proc 5279252481
[cn1:14034] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 5279252481
[cn1:14034] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 5279252481
[cn1:14034] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 5279252481
[cn1:14034] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 5279252481
[cn1:14034] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 5279252481
[cn1:14034] [[15019,1],1] grpcomm:base:pack_modex: reporting 5 entries
[cn1:14034] [[15019,1],1] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn1:14034] [[15019,1],1] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn1:14034] [[15019,1],1] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn1:14034] [[15019,1],1] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn1:14034] [[15019,1],1] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn1:14034] [[15019,1],1] grpcomm:base:full:modex: executing allgather
[cn1:14034] [[15019,1],1] grpcomm:bad entering allgather
[cn1:14034] [[15019,1],1] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn1:14034] [[15019,1],1] grpcomm:base:modex: modex posted
[cn1:14034] [[15019,1],1] grpcomm:bad sending collective 0 to our daemon
[cn1:14034] [[15019,1],1] rml_send_buffer to peer [[15019,0],0] at tag 30
[cn1:14034] [[15019,1],1] grpcomm:bad allgather underway
[cn1:14034] [[15019,1],1] rml_send_msg to peer [[15019,0],0] at tag 30
[cn1:14034] [[15019,1],1] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14034] [[15019,1],1] oob:base:send to target [[15019,0],0]
[cn1:14034] [[15019,1],1] oob:base:send known transport for peer [[15019,0],0]
[cn1:14034] [[15019,1],1] oob:tcp:send_nb to peer [[15019,0],0]:30
[cn1:14034] [[15019,1],1] tcp:send_nb to peer [[15019,0],0]
[cn1:14034] [[15019,1],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14034] [[15019,1],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:30
[cn1:14034] [[15019,1],1] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14034] [[15019,1],1] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14034] [[15019,1],1]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14034] [[15019,1],1] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14034] [[15019,1],1] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14034] [[15019,1],1] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 372 BYTES ON SOCKET 11
[cn1:14034] [[15019,1],1]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14033] [[15019,1],0] grpcomm:base:modex: performing modex
[cn1:14033] fetching data from PEER dstore
[cn1:14033] dstore:hash:fetch: searching for key NULL on proc 984285185
[cn1:14033] dstore:hash:fetch: adding data for key MPI_THREAD_LEVEL on proc 984285185
[cn1:14033] dstore:hash:fetch: adding data for key btl.openib.1.9 on proc 984285185
[cn1:14033] dstore:hash:fetch: adding data for key btl.tcp.1.9 on proc 984285185
[cn1:14033] dstore:hash:fetch: adding data for key btl.usnic.1.9 on proc 984285185
[cn1:14033] dstore:hash:fetch: adding data for key btl.vader.1.9 on proc 984285185
[cn1:14033] [[15019,1],0] grpcomm:base:pack_modex: reporting 5 entries
[cn1:14033] [[15019,1],0] grpcomm:base:pack_modex: packing entry for MPI_THREAD_LEVEL
[cn1:14033] [[15019,1],0] grpcomm:base:pack_modex: packing entry for btl.openib.1.9
[cn1:14033] [[15019,1],0] grpcomm:base:pack_modex: packing entry for btl.tcp.1.9
[cn1:14033] [[15019,1],0] grpcomm:base:pack_modex: packing entry for btl.usnic.1.9
[cn1:14033] [[15019,1],0] grpcomm:base:pack_modex: packing entry for btl.vader.1.9
[cn1:14033] [[15019,1],0] grpcomm:base:full:modex: executing allgather
[cn1:14033] [[15019,1],0] grpcomm:bad entering allgather
[cn1:14033] [[15019,1],0] ACTIVATING GRCPCOMM OP 0 at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/grpcomm/bad/grpcomm_bad_module.c:315
[cn1:14033] [[15019,1],0] grpcomm:base:modex: modex posted
[cn1:14033] [[15019,1],0] grpcomm:bad sending collective 0 to our daemon
[cn1:14033] [[15019,1],0] rml_send_buffer to peer [[15019,0],0] at tag 30
[cn1:14033] [[15019,1],0] grpcomm:bad allgather underway
[cn1:14033] [[15019,1],0] rml_send_msg to peer [[15019,0],0] at tag 30
[cn1:14033] [[15019,1],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14033] [[15019,1],0] oob:base:send to target [[15019,0],0]
[cn1:14033] [[15019,1],0] oob:base:send known transport for peer [[15019,0],0]
[cn1:14033] [[15019,1],0] oob:tcp:send_nb to peer [[15019,0],0]:30
[cn1:14033] [[15019,1],0] tcp:send_nb to peer [[15019,0],0]
[cn1:14033] [[15019,1],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],0]
[cn1:14033] [[15019,1],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],0]:30
[cn1:14033] [[15019,1],0] routed_binomial_get([[15019,0],0]) --> [[15019,0],0]
[cn1:14033] [[15019,1],0] tcp:send_nb: already connected to [[15019,0],0] - queueing for send
[cn1:14033] [[15019,1],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,1],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate data region of size 372
[cn1:14033] [[15019,1],0] tcp:send_handler called to send to peer [[15019,0],0]
[cn1:14033] [[15019,1],0] tcp:send_handler SENDING TO [[15019,0],0]
[cn1:14033] [[15019,1],0] MESSAGE SEND COMPLETE TO [[15019,0],0] OF 372 BYTES ON SOCKET 11
[cn1:14033] [[15019,1],0]-[[15019,0],0] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,1],1]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,1],1] (ORIGIN [[15019,1],1]) OF 372 BYTES FOR DEST [[15019,0],0] TAG 30
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 372 bytes from [[15019,1],1] for tag 30
[cn1:14018] [[15019,0],0] COLLECTIVE RECVD FROM [[15019,1],1]
[cn1:14018] [[15019,0],0] WORKING COLLECTIVE 0
[cn1:14018] [[15019,0],0] PROGRESSING COLLECTIVE 0
[cn1:14018] [[15019,0],0] PROGRESSING COLL id 0
[cn1:14018] [[15019,0],0] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn1:14018] [[15019,0],0] PROGRESSING COLL id 1
[cn1:14018] [[15019,0],0] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn1:14018] [[15019,0],0] PROGRESSING COLL id 2
[cn1:14018] [[15019,0],0] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,1],0]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0] RECVD COMPLETE MESSAGE FROM [[15019,1],0] (ORIGIN [[15019,1],0]) OF 372 BYTES FOR DEST [[15019,0],0] TAG 30
[cn1:14018] [[15019,0],0] DELIVERING TO RML
[cn1:14018] [[15019,0],0] Message posted at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:536
[cn1:14018] [[15019,0],0] message received 372 bytes from [[15019,1],0] for tag 30
[cn1:14018] [[15019,0],0] COLLECTIVE RECVD FROM [[15019,1],0]
[cn1:14018] [[15019,0],0] WORKING COLLECTIVE 0
[cn1:14018] [[15019,0],0] PROGRESSING COLLECTIVE 0
[cn1:14018] [[15019,0],0] PROGRESSING COLL id 0
[cn1:14018] [[15019,0],0] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn1:14018] [[15019,0],0] COLLECTIVE 0 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],0] at tag 32
[cn1:14018] [[15019,0],0] PROGRESSING COLL id 1
[cn1:14018] [[15019,0],0] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn1:14018] [[15019,0],0] PROGRESSING COLL id 2
[cn1:14018] [[15019,0],0] ALL LOCAL PROCS FOR JOB [15019,1] CONTRIBUTE 2
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],0] at tag 32
[cn1:14018] [[15019,0],0] rml_send_iovec_to_self at tag 32
[cn1:14018] [[15019,0],0] rml_send_to_self callback executing for tag 32
[cn1:14018] [[15019,0],0] message received 756 bytes from [[15019,0],0] for tag 32
[cn1:14018] [[15019,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[15019,0],0]
[cn1:14018] [[15019,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0
[cn1:14018] [[15019,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 2
[cn1:14018] [[15019,0],0] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 10 ngr 8
srun: Job step aborted: Waiting up to 2 seconds for job step to finish.
mpirun: Forwarding signal 18 to job
[cn1:14018] [[15019,0],0] plm:base:orted_cmd sending signal_local_procs cmds
[cn1:14018] [[15019,0],0] grpcomm:bad:xcast sent to job [15019,0] tag 1
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],0] at tag 15
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],0] at tag 15
[cn1:14018] [[15019,0],0] rml_send_iovec_to_self at tag 15
[cn1:14018] [[15019,0],0] rml_send_to_self callback executing for tag 15
[cn1:14018] [[15019,0],0] message received 29 bytes from [[15019,0],0] for tag 15
[cn1:14018] [[15019,0],0] grpcomm:xcast:recv: with 29 bytes
[cn1:14018] [[15019,0],0] orte:daemon:send_relay sending relay msg to [[15019,0],1]
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],1] at tag 15
[cn1:14018] [[15019,0],0] orte:daemon:send_relay sending relay msg to [[15019,0],2]
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],2] at tag 15
[cn1:14018] [[15019,0],0] orte:daemon:send_relay sending relay msg to [[15019,0],4]
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],4] at tag 15
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],0] at tag 1
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],1] at tag 15
[cn1:14018] [[15019,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],2] at tag 15
[cn1:14018] [[15019,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],4] at tag 15
[cn1:14018] [[15019,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],0] at tag 1
[cn1:14018] [[15019,0],0] rml_send_iovec_to_self at tag 1
[cn1:14018] [[15019,0],0] oob:base:send to target [[15019,0],1]
[cn1:14018] [[15019,0],0] oob:base:send known transport for peer [[15019,0],1]
[cn1:14018] [[15019,0],0] oob:tcp:send_nb to peer [[15019,0],1]:15
[cn1:14018] [[15019,0],0] tcp:send_nb to peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],1]
[cn1:14018] [[15019,0],0] oob:base:send to target [[15019,0],2]
[cn1:14018] [[15019,0],0] oob:base:send known transport for peer [[15019,0],2]
[cn1:14018] [[15019,0],0] oob:tcp:send_nb to peer [[15019,0],2]:15
[cn1:14018] [[15019,0],0] tcp:send_nb to peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],2]
[cn1:14018] [[15019,0],0] oob:base:send to target [[15019,0],4]
[cn1:14018] [[15019,0],0] oob:base:send known transport for peer [[15019,0],4]
[cn1:14018] [[15019,0],0] oob:tcp:send_nb to peer [[15019,0],4]:15
[cn1:14018] [[15019,0],0] tcp:send_nb to peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],4]
[cn1:14018] [[15019,0],0] rml_send_to_self callback executing for tag 1
[cn1:14018] [[15019,0],0] message received 29 bytes from [[15019,0],0] for tag 1
[cn1:14018] [[15019,0],0] orted:comm:process_commands() Processing Command: ORTE_DAEMON_SIGNAL_LOCAL_PROCS
[cn1:14018] [[15019,0],0] orted_cmd: received signal_local_procs, delivering signal 18
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],1]:15
[cn1:14018] [[15019,0],0] routed_binomial_get([[15019,0],1]) --> [[15019,0],1]
[cn1:14018] [[15019,0],0] tcp:send_nb: already connected to [[15019,0],1] - queueing for send
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],1]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],2]:15
[cn1:14018] [[15019,0],0] routed_binomial_get([[15019,0],2]) --> [[15019,0],2]
[cn1:14018] [[15019,0],0] tcp:send_nb: already connected to [[15019,0],2] - queueing for send
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],2]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:442] processing send to peer [[15019,0],4]:15
[cn1:14018] [[15019,0],0] routed_binomial_get([[15019,0],4]) --> [[15019,0],4]
[cn1:14018] [[15019,0],0] tcp:send_nb: already connected to [[15019,0],4] - queueing for send
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:469] queue send to [[15019,0],4]
[cn1:14018] [[15019,0],0] tcp:send_handler called to send to peer [[15019,0],4]
[cn1:14018] [[15019,0],0] tcp:send_handler SENDING TO [[15019,0],4]
[cn1:14018] [[15019,0],0] MESSAGE SEND COMPLETE TO [[15019,0],4] OF 29 BYTES ON SOCKET 14
[cn1:14018] [[15019,0],0]-[[15019,0],4] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] tcp:send_handler called to send to peer [[15019,0],1]
[cn1:14018] [[15019,0],0] tcp:send_handler SENDING TO [[15019,0],1]
[cn1:14018] [[15019,0],0] MESSAGE SEND COMPLETE TO [[15019,0],1] OF 29 BYTES ON SOCKET 15
[cn1:14018] [[15019,0],0]-[[15019,0],1] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn1:14018] [[15019,0],0] tcp:send_handler called to send to peer [[15019,0],2]
[cn1:14018] [[15019,0],0] tcp:send_handler SENDING TO [[15019,0],2]
slurmd[cn2]: *** STEP 3521.0 CANCELLED AT 2014-06-01T13:37:46 ***
[cn1:14018] [[15019,0],0] MESSAGE SEND COMPLETE TO [[15019,0],2] OF 29 BYTES ON SOCKET 17
[cn1:14018] [[15019,0],0]-[[15019,0],2] Send message complete at /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp_sendrecv.c:224
[cn2:00814] [[15019,0],1]:/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/ess/base/ess_base_std_orted.c(711) updating exit status to 1
[cn5:14014] [[15019,0],4]:tcp:recv:handler called for peer [[15019,0],0]
[cn5:14014] [[15019,0],4]:tcp:recv:handler CONNECTED
[cn5:14014] [[15019,0],4]:tcp:recv:handler allocate new recv msg
[cn5:14014] [[15019,0],4]:tcp:recv:handler read hdr
[cn5:14014] [[15019,0],4]:tcp:recv:handler allocate data region of size 29
[cn1:14018] [[15019,0],0]:tcp:recv:handler called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0]:tcp:recv:handler CONNECTED
[cn1:14018] [[15019,0],0]:tcp:recv:handler allocate new recv msg
srun: error: cn5: task 3: Exited with exit code 1
[cn1:14018] [[15019,0],0]:tcp:recv:handler read hdr
[cn1:14018] [[15019,0],0]-[[15019,0],4] mca_oob_tcp_msg_recv: peer closed connection
slurmd[cn1]: *** JOB 3521 CANCELLED AT 2014-06-01T13:37:46 ***
[cn1:14018] [[15019,0],0] tcp_peer_close for [[15019,0],4] sd 14 state CONNECTED
[cn1:14018] [[15019,0],0] tcp:lost connection called for peer [[15019,0],4]
[cn1:14018] [[15019,0],0] route to [[15019,0],4] lost
[cn1:14018] [[15019,0],0] routed_binomial: removing route to child daemon [[15019,0],4]
[cn1:14018] [[15019,0],0] plm:base:orted_cmd sending orted_exit commands
[cn1:14018] [[15019,0],0] grpcomm:bad:xcast sent to job [15019,0] tag 1
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],0] at tag 15
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],0] at tag 15
[cn1:14018] [[15019,0],0] rml_send_iovec_to_self at tag 15
[cn1:14018] [[15019,0],0] rml_send_to_self callback executing for tag 15
[cn1:14018] [[15019,0],0] message received 8 bytes from [[15019,0],0] for tag 15
[cn1:14018] [[15019,0],0] grpcomm:xcast:recv: with 8 bytes
[cn1:14018] [[15019,0],0] orte:daemon:send_relay sending relay msg to [[15019,0],1]
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],1] at tag 15
[cn1:14018] [[15019,0],0] orte:daemon:send_relay sending relay msg to [[15019,0],2]
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],2] at tag 15
[cn1:14018] [[15019,0],0] orte:daemon:send_relay sending relay msg to [[15019,0],3]
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],3] at tag 15
[cn1:14018] [[15019,0],0] rml_send_buffer to peer [[15019,0],0] at tag 1
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],1] at tag 15
[cn1:14018] [[15019,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],2] at tag 15
[cn1:14018] [[15019,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
srun: Terminating job step 3521.0
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],3] at tag 15
[cn1:14018] [[15019,0],0] OOB_SEND: /home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/rml/oob/rml_oob_send.c:199
[cn1:14018] [[15019,0],0] rml_send_msg to peer [[15019,0],0] at tag 1
[cn1:14018] [[15019,0],0] rml_send_iovec_to_self at tag 1
[cn1:14018] [[15019,0],0] oob:base:send to target [[15019,0],1]
[cn1:14018] [[15019,0],0] oob:base:send known transport for peer [[15019,0],1]
[cn1:14018] [[15019,0],0] oob:tcp:send_nb to peer [[15019,0],1]:15
[cn1:14018] [[15019,0],0] tcp:send_nb to peer [[15019,0],1]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],1]
[cn1:14018] [[15019,0],0] oob:base:send to target [[15019,0],2]
[cn1:14018] [[15019,0],0] oob:base:send known transport for peer [[15019,0],2]
[cn1:14018] [[15019,0],0] oob:tcp:send_nb to peer [[15019,0],2]:15
[cn1:14018] [[15019,0],0] tcp:send_nb to peer [[15019,0],2]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],2]
[cn1:14018] [[15019,0],0] oob:base:send to target [[15019,0],3]
[cn1:14018] [[15019,0],0] oob:base:send known transport for peer [[15019,0],3]
[cn1:14018] [[15019,0],0] oob:tcp:send_nb to peer [[15019,0],3]:15
[cn1:14018] [[15019,0],0] tcp:send_nb to peer [[15019,0],3]
[cn1:14018] [[15019,0],0]:[/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/oob/tcp/oob_tcp.c:508] post send to [[15019,0],3]
[cn5:14014] [[15019,0],4]:/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/ess/base/ess_base_std_orted.c(711) updating exit status to 1
[cn1:14018] [[15019,0],0] rml_send_to_self callback executing for tag 1
[cn1:14018] [[15019,0],0] message received 8 bytes from [[15019,0],0] for tag 1
[cn1:14018] [[15019,0],0] orted:comm:process_commands() Processing Command: Unknown Command!
[cn4:17677] [[15019,0],3]:/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/ess/base/ess_base_std_orted.c(711) updating exit status to 1
[cn1:14018] [[15019,0],0] orted_cmd: received halt_vm cmd
[cn3:21265] [[15019,0],2]:tcp:recv:handler called for peer [[15019,0],0]
[cn3:21265] [[15019,0],2]:tcp:recv:handler CONNECTED
[cn3:21265] [[15019,0],2]:tcp:recv:handler allocate new recv msg
[cn3:21265] [[15019,0],2]:tcp:recv:handler read hdr
[cn3:21265] [[15019,0],2]:tcp:recv:handler allocate data region of size 29
[cn2:00814] sess_dir_finalize: proc session dir does not exist
[cn5:14014] sess_dir_finalize: proc session dir does not exist
[cn4:17677] sess_dir_finalize: proc session dir does not exist
[cn3:21265] [[15019,0],2]:/home/research/artpol/ompi_dev//ompi-trunk_r31907/orte/mca/ess/base/ess_base_std_orted.c(711) updating exit status to 1
[cn2:00814] sess_dir_finalize: proc session dir does not exist
[cn2:00814] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn2:00814] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn2:00814] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn5:14014] sess_dir_finalize: proc session dir does not exist
[cn5:14014] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn5:14014] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn5:14014] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn4:17677] sess_dir_finalize: proc session dir does not exist
[cn4:17677] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn4:17677] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn4:17677] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn3:21265] sess_dir_finalize: proc session dir does not exist
[cn2:00814] [[15019,0],1] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 21
[cn5:14014] [[15019,0],4] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 21
[cn4:17677] [[15019,0],3] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 21
[cn3:21265] sess_dir_finalize: proc session dir does not exist
[cn3:21265] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn3:21265] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn3:21265] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn2:00814] [[15019,0],1] grpcomm:base:receive stop comm
[cn2:00814] [[15019,0],1] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 30
[cn2:00814] [[15019,0],1] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 15
[cn2:00814] [[15019,0],1] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 32
[cn2:00814] mca: base: close: component bad closed
[cn2:00814] mca: base: close: unloading component bad
[cn5:14014] [[15019,0],4] grpcomm:base:receive stop comm
[cn5:14014] [[15019,0],4] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 30
[cn5:14014] [[15019,0],4] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 15
[cn5:14014] [[15019,0],4] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 32
[cn5:14014] mca: base: close: component bad closed
[cn5:14014] mca: base: close: unloading component bad
[cn4:17677] [[15019,0],3] grpcomm:base:receive stop comm
[cn4:17677] [[15019,0],3] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 30
[cn3:21265] [[15019,0],2] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 21
[cn2:00814] [[15019,0],1] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 3
[cn5:14014] [[15019,0],4] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 3
[cn4:17677] [[15019,0],3] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 15
[cn4:17677] [[15019,0],3] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 32
[cn4:17677] mca: base: close: component bad closed
[cn4:17677] mca: base: close: unloading component bad
[cn3:21265] [[15019,0],2] grpcomm:base:receive stop comm
[cn3:21265] [[15019,0],2] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 30
[cn3:21265] [[15019,0],2] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 15
[cn3:21265] [[15019,0],2] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 32
[cn3:21265] mca: base: close: component bad closed
[cn3:21265] mca: base: close: unloading component bad
[cn2:00814] [[15019,0],1] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 45
[cn2:00814] [[15019,0],1] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 46
[cn5:14014] [[15019,0],4] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 45
[cn5:14014] [[15019,0],4] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 46
[cn4:17677] [[15019,0],3] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 3
[cn3:21265] [[15019,0],2] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 3
[cn2:00814] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn2:00814] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn2:00814] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn5:14014] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn5:14014] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn5:14014] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn4:17677] [[15019,0],3] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 45
[cn3:21265] [[15019,0],2] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 45
[cn2:00814] mca: base: close: unloading component binomial
[cn5:14014] mca: base: close: unloading component binomial
[cn4:17677] [[15019,0],3] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 46
[cn3:21265] [[15019,0],2] rml_recv_cancel for peer [[WILDCARD],WILDCARD] tag 46
[cn2:00814] mca: base: close: component oob closed
[cn2:00814] mca: base: close: unloading component oob
[cn5:14014] mca: base: close: component oob closed
[cn5:14014] mca: base: close: unloading component oob
[cn4:17677] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn4:17677] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn4:17677] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn3:21265] Waiting for thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn3:21265] Thread obtained /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:118
[cn2:00814] [[15019,0],1] TCP SHUTDOWN
[cn5:14014] [[15019,0],4] TCP SHUTDOWN
[cn4:17677] mca: base: close: unloading component binomial
[cn4:17677] mca: base: close: component oob closed
[cn4:17677] mca: base: close: unloading component oob
[cn3:21265] Releasing thread /home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/class/opal_ring_buffer.c:135
[cn2:00814] [[15019,0],1] RELEASING PEER OBJ [[15019,0],0]
[cn2:00814] [[15019,0],1] CLOSING SOCKET 9
[cn5:14014] [[15019,0],4] RELEASING PEER OBJ [[15019,0],0]
[cn5:14014] [[15019,0],4] CLOSING SOCKET 9
[cn4:17677] [[15019,0],3] TCP SHUTDOWN
[cn3:21265] mca: base: close: unloading component binomial
[cn2:00814] [[15019,0],1] RELEASING PEER OBJ [[15019,0],2]
[cn2:00814] [[15019,0],1] RELEASING PEER OBJ [[15019,0],3]
[cn5:14014] [[15019,0],4] RELEASING PEER OBJ [[15019,0],1]
[cn4:17677] [[15019,0],3] RELEASING PEER OBJ [[15019,0],0]
[cn4:17677] [[15019,0],3] CLOSING SOCKET 9
[cn3:21265] mca: base: close: component oob closed
[cn3:21265] mca: base: close: unloading component oob
[cn2:00814] [[15019,0],1] CLOSING SOCKET 10
[cn5:14014] [[15019,0],4] RELEASING PEER OBJ [[15019,0],2]
[cn5:14014] [[15019,0],4] RELEASING PEER OBJ [[15019,0],3]
[cn5:14014] [[15019,0],4] RELEASING PEER OBJ [[15019,1],8]
[cn4:17677] [[15019,0],3] RELEASING PEER OBJ [[15019,0],1]
[cn4:17677] [[15019,0],3] CLOSING SOCKET 10
[cn3:21265] [[15019,0],2] TCP SHUTDOWN
[cn2:00814] [[15019,0],1] RELEASING PEER OBJ [[15019,0],4]
[cn2:00814] [[15019,0],1] RELEASING PEER OBJ [[15019,1],2]
[cn5:14014] [[15019,0],4] CLOSING SOCKET 12
[cn4:17677] [[15019,0],3] RELEASING PEER OBJ [[15019,0],2]
[cn3:21265] [[15019,0],2] RELEASING PEER OBJ [[15019,0],0]
[cn3:21265] [[15019,0],2] CLOSING SOCKET 9
[cn2:00814] [[15019,0],1] CLOSING SOCKET 13
[cn5:14014] [[15019,0],4] RELEASING PEER OBJ [[15019,1],9]
[cn5:14014] [[15019,0],4] CLOSING SOCKET 13
[cn4:17677] [[15019,0],3] RELEASING PEER OBJ [[15019,0],4]
[cn4:17677] [[15019,0],3] RELEASING PEER OBJ [[15019,1],6]
[cn4:17677] [[15019,0],3] CLOSING SOCKET 13
[cn3:21265] [[15019,0],2] RELEASING PEER OBJ [[15019,0],1]
[cn2:00814] [[15019,0],1] RELEASING PEER OBJ [[15019,1],3]
[cn2:00814] [[15019,0],1] CLOSING SOCKET 14
[cn5:14014] mca: base: close: component tcp closed
[cn5:14014] mca: base: close: unloading component tcp
[cn4:17677] [[15019,0],3] RELEASING PEER OBJ [[15019,1],7]
[cn4:17677] [[15019,0],3] CLOSING SOCKET 14
[cn3:21265] [[15019,0],2] RELEASING PEER OBJ [[15019,0],3]
[cn3:21265] [[15019,0],2] RELEASING PEER OBJ [[15019,0],4]
[cn2:00814] mca: base: close: component tcp closed
[cn2:00814] mca: base: close: unloading component tcp
[cn5:14014] mca: base: close: unloading component hash
[cn4:17677] mca: base: close: component tcp closed
[cn4:17677] mca: base: close: unloading component tcp
[cn3:21265] [[15019,0],2] RELEASING PEER OBJ [[15019,1],4]
[cn3:21265] [[15019,0],2] CLOSING SOCKET 12
[cn2:00814] mca: base: close: unloading component hash
[cn5:14014] sess_dir_cleanup: job session dir does not exist
[cn4:17677] mca: base: close: unloading component hash
[cn3:21265] [[15019,0],2] RELEASING PEER OBJ [[15019,1],5]
[cn2:00814] sess_dir_cleanup: job session dir does not exist
[cn5:14014] mca: base: close: component slurm closed
[cn5:14014] mca: base: close: unloading component slurm
[cn4:17677] sess_dir_cleanup: job session dir does not exist
[cn3:21265] [[15019,0],2] CLOSING SOCKET 13
[cn2:00814] mca: base: close: component slurm closed
[cn2:00814] mca: base: close: unloading component slurm
exiting with status 1
[cn4:17677] mca: base: close: component slurm closed
[cn4:17677] mca: base: close: unloading component slurm
[cn3:21265] mca: base: close: component tcp closed
[cn3:21265] mca: base: close: unloading component tcp
exiting with status 1
exiting with status 1
[cn3:21265] mca: base: close: unloading component hash
[cn3:21265] sess_dir_cleanup: job session dir does not exist
[cn3:21265] mca: base: close: component slurm closed
[cn3:21265] mca: base: close: unloading component slurm
exiting with status 1
srun: error: cn2: task 0: Exited with exit code 1
srun: error: cn3: task 1: Exited with exit code 1
srun: error: cn4: task 2: Exited with exit code 1
