Ok, I figured out that it was not a problem with the node grsacc04 because I
now conducted the same on totally different set of nodes.
I must really say that with --bind-to none option, the program completed "many"
times compared to earlier but still "sometimes" it hangs! Attaching now the
output of the same case conducted on different set of nodes with the --bind-to
none option.
mpiexec -mca plm_base_verbose 5 -mca ess_base_verbose 5 -mca
grpcomm_base_verbose 5 --bind-to none -np 3 ./example
Best,
Suraj
{\rtf1\ansi\ansicpg1252\cocoartf1038\cocoasubrtf360
{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
{\colortbl;\red255\green255\blue255;}
\paperw11900\paperh16840\margl1440\margr1440\vieww31940\viewh20120\viewkind0
\pard\tx566\tx1133\tx1700\tx2267\tx2834\tx3401\tx3968\tx4535\tx5102\tx5669\tx6236\tx6803\ql\qnatural\pardirnatural
\f0\fs24 \cf0 [grsacc06:19901] mca:base:select:( ess) Querying component [env]\
[grsacc06:19901] mca:base:select:( ess) Skipping component [env]. Query failed to return a module\
[grsacc06:19901] mca:base:select:( ess) Querying component [hnp]\
[grsacc06:19901] mca:base:select:( ess) Query of component [hnp] set priority to 100\
[grsacc06:19901] mca:base:select:( ess) Querying component [singleton]\
[grsacc06:19901] mca:base:select:( ess) Skipping component [singleton]. Query failed to return a module\
[grsacc06:19901] mca:base:select:( ess) Querying component [slurm]\
[grsacc06:19901] mca:base:select:( ess) Skipping component [slurm]. Query failed to return a module\
[grsacc06:19901] mca:base:select:( ess) Querying component [tm]\
[grsacc06:19901] mca:base:select:( ess) Skipping component [tm]. Query failed to return a module\
[grsacc06:19901] mca:base:select:( ess) Querying component [tool]\
[grsacc06:19901] mca:base:select:( ess) Skipping component [tool]. Query failed to return a module\
[grsacc06:19901] mca:base:select:( ess) Selected component [hnp]\
[grsacc06:19901] [[INVALID],INVALID] Topology Info:\
[grsacc06:19901] Type: Machine Number of child objects: 3\
Name=NULL\
total=25156656KB\
DMIProductName=X8DTT-H\
DMIProductVersion=1234567890\
DMIBoardVendor=Supermicro\
DMIBoardName=X8DTT-H\
DMIBoardVersion=1234567890\
DMIBoardAssetTag=1234567890\
DMIChassisVendor=Supermicro\
DMIChassisType=17\
DMIChassisVersion=1234567890\
DMIChassisAssetTag="To Be Filled By O.E.M."\
DMIBIOSVendor="American Megatrends Inc."\
DMIBIOSVersion="080015 "\
DMIBIOSDate=12/10/2009\
DMISysVendor=Supermicro\
Backend=Linux\
OSName=Linux\
OSRelease=2.6.35-30-generic\
OSVersion="#56-Ubuntu SMP Mon Jul 11 20:01:08 UTC 2011"\
Architecture=x86_64\
Cpuset: 0x000000ff\
Online: 0x000000ff\
Allowed: 0x000000ff\
Bind CPU proc: TRUE\
Bind CPU thread: TRUE\
Bind MEM proc: FALSE\
Bind MEM thread: TRUE\
Type: NUMANode Number of child objects: 1\
Name=NULL\
local=12573744KB\
total=12573744KB\
Cpuset: 0x0000000f\
Online: 0x0000000f\
Allowed: 0x0000000f\
Type: Socket Number of child objects: 1\
Name=NULL\
CPUModel="Intel(R) Xeon(R) CPU X5570 @ 2.93GHz"\
Cpuset: 0x0000000f\
Online: 0x0000000f\
Allowed: 0x0000000f\
Type: L3Cache Number of child objects: 4\
Name=NULL\
size=8192KB\
linesize=64\
ways=16\
Cpuset: 0x0000000f\
Online: 0x0000000f\
Allowed: 0x0000000f\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000002\
Online: 0x00000002\
Allowed: 0x00000002\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000002\
Online: 0x00000002\
Allowed: 0x00000002\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000002\
Online: 0x00000002\
Allowed: 0x00000002\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000002\
Online: 0x00000002\
Allowed: 0x00000002\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: NUMANode Number of child objects: 1\
Name=NULL\
local=12582912KB\
total=12582912KB\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: Socket Number of child objects: 1\
Name=NULL\
CPUModel="Intel(R) Xeon(R) CPU X5570 @ 2.93GHz"\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: L3Cache Number of child objects: 4\
Name=NULL\
size=8192KB\
linesize=64\
ways=16\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x00000040\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x00000040\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x00000040\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x00000040\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: Bridge Host->PCI Number of child objects: 5\
Name=NULL\
buses=0000:[00-06]\
Type: PCI 8086:10d3 Number of child objects: 1\
Name=Intel Corporation 82574L Gigabit Network Connection\
busid=0000:01:00.0\
class=0200(Ether)\
PCIVendor="Intel Corporation"\
PCIDevice="82574L Gigabit Network Connection"\
Type: Network Number of child objects: 0\
Name=eth0\
Address=00:30:48:cf:63:b8\
Type: PCI 8086:10d3 Number of child objects: 1\
Name=Intel Corporation 82574L Gigabit Network Connection\
busid=0000:02:00.0\
class=0200(Ether)\
PCIVendor="Intel Corporation"\
PCIDevice="82574L Gigabit Network Connection"\
Type: Network Number of child objects: 0\
Name=eth1\
Address=00:30:48:cf:63:b9\
Type: PCI 15b3:673c Number of child objects: 2\
Name=Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]\
busid=0000:03:00.0\
class=0c06(IB)\
PCIVendor="Mellanox Technologies"\
PCIDevice="MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]"\
Type: Network Number of child objects: 0\
Name=ib0\
Address=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:ce:0f:c9\
Port=1\
Type: OpenFabrics Number of child objects: 0\
Name=mlx4_0\
NodeGUID=0030:48ff:ffce:0fc8\
SysImageGUID=0030:48ff:ffce:0fcb\
Port1State=4\
Port1LID=0x5\
Port1LMC=0\
Port1GID0=fe80:0000:0000:0000:0030:48ff:ffce:0fc9\
Type: PCI 102b:0532 Number of child objects: 0\
Name=Matrox Graphics, Inc. MGA G200eW WPCM450\
busid=0000:06:01.0\
class=0300(VGA)\
PCIVendor="Matrox Graphics, Inc."\
PCIDevice="MGA G200eW WPCM450"\
Type: PCI 8086:3a20 Number of child objects: 1\
Name=Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1\
busid=0000:00:1f.2\
class=0101(IDE)\
PCIVendor="Intel Corporation"\
PCIDevice="82801JI (ICH10 Family) 4 port SATA IDE Controller #1"\
Type: Block Number of child objects: 0\
Name=sda\
[grsacc06:19901] mca:base:select:( plm) Querying component [rsh]\
[grsacc06:19901] [[INVALID],INVALID] plm:rsh_lookup on agent ssh : rsh path NULL\
[grsacc06:19901] mca:base:select:( plm) Query of component [rsh] set priority to 10\
[grsacc06:19901] mca:base:select:( plm) Querying component [slurm]\
[grsacc06:19901] mca:base:select:( plm) Skipping component [slurm]. Query failed to return a module\
[grsacc06:19901] mca:base:select:( plm) Querying component [tm]\
[grsacc06:19901] mca:base:select:( plm) Query of component [tm] set priority to 75\
[grsacc06:19901] mca:base:select:( plm) Selected component [tm]\
[grsacc06:19901] plm:base:set_hnp_name: initial bias 19901 nodename hash 2587242904\
[grsacc06:19901] plm:base:set_hnp_name: final jobfam 58899\
[grsacc06:19901] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc06:19901] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc06:19901] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:receive start comm\
[grsacc06:19901] [[58899,0],0] plm:base:receive start comm\
[grsacc06:19901] [[58899,0],0] plm:base:setup_job\
[grsacc06:19901] [[58899,0],0] plm:base:setup_vm\
[grsacc06:19901] [[58899,0],0] plm:base:setup_vm creating map\
[grsacc06:19901] [[58899,0],0] plm:base:setup_vm add new daemon [[58899,0],1]\
[grsacc06:19901] [[58899,0],0] plm:base:setup_vm assigning new daemon [[58899,0],1] to node grsacc07\
[grsacc06:19901] [[58899,0],0] plm:base:setup_vm add new daemon [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] plm:base:setup_vm assigning new daemon [[58899,0],2] to node grsacc08\
[grsacc06:19901] [[58899,0],0] plm:base:setup_vm add new daemon [[58899,0],3]\
[grsacc06:19901] [[58899,0],0] plm:base:setup_vm assigning new daemon [[58899,0],3] to node grsacc09\
[grsacc06:19901] [[58899,0],0] plm:base:setup_vm add new daemon [[58899,0],4]\
[grsacc06:19901] [[58899,0],0] plm:base:setup_vm assigning new daemon [[58899,0],4] to node grsacc10\
[grsacc06:19901] [[58899,0],0] plm:tm: launching vm\
[grsacc06:19901] [[58899,0],0] plm:tm: final top-level argv:\
orted -mca ess tm -mca orte_ess_jobid 3860004864 -mca orte_ess_vpid <template> -mca orte_ess_num_procs 5 -mca orte_hnp_uri "3860004864.0;tcp://192.168.222.6,192.168.111.6:53034" -mca plm_base_verbose 5 -mca ess_base_verbose 5 -mca grpcomm_base_verbose 5\
[grsacc06:19901] [[58899,0],0] plm:tm: launching on node grsacc07\
[grsacc06:19901] [[58899,0],0] plm:tm: executing:\
orted -mca ess tm -mca orte_ess_jobid 3860004864 -mca orte_ess_vpid 1 -mca orte_ess_num_procs 5 -mca orte_hnp_uri "3860004864.0;tcp://192.168.222.6,192.168.111.6:53034" -mca plm_base_verbose 5 -mca ess_base_verbose 5 -mca grpcomm_base_verbose 5\
[grsacc06:19901] [[58899,0],0] plm:tm: launching on node grsacc08\
[grsacc06:19901] [[58899,0],0] plm:tm: executing:\
orted -mca ess tm -mca orte_ess_jobid 3860004864 -mca orte_ess_vpid 2 -mca orte_ess_num_procs 5 -mca orte_hnp_uri "3860004864.0;tcp://192.168.222.6,192.168.111.6:53034" -mca plm_base_verbose 5 -mca ess_base_verbose 5 -mca grpcomm_base_verbose 5\
[grsacc06:19901] [[58899,0],0] plm:tm: launching on node grsacc09\
[grsacc06:19901] [[58899,0],0] plm:tm: executing:\
orted -mca ess tm -mca orte_ess_jobid 3860004864 -mca orte_ess_vpid 3 -mca orte_ess_num_procs 5 -mca orte_hnp_uri "3860004864.0;tcp://192.168.222.6,192.168.111.6:53034" -mca plm_base_verbose 5 -mca ess_base_verbose 5 -mca grpcomm_base_verbose 5\
[grsacc06:19901] [[58899,0],0] plm:tm: launching on node grsacc10\
[grsacc06:19901] [[58899,0],0] plm:tm: executing:\
orted -mca ess tm -mca orte_ess_jobid 3860004864 -mca orte_ess_vpid 4 -mca orte_ess_num_procs 5 -mca orte_hnp_uri "3860004864.0;tcp://192.168.222.6,192.168.111.6:53034" -mca plm_base_verbose 5 -mca ess_base_verbose 5 -mca grpcomm_base_verbose 5\
[grsacc06:19901] [[58899,0],0] plm:tm:launch: finished spawning orteds\
[grsacc07:01601] mca:base:select:( ess) Querying component [tm]\
[grsacc07:01601] mca:base:select:( ess) Query of component [tm] set priority to 30\
[grsacc07:01601] mca:base:select:( ess) Selected component [tm]\
[grsacc07:01601] ess:tm setting name\
[grsacc07:01601] ess:tm set name to [[58899,0],1]\
[grsacc08:09178] mca:base:select:( ess) Querying component [tm]\
[grsacc08:09178] mca:base:select:( ess) Query of component [tm] set priority to 30\
[grsacc08:09178] mca:base:select:( ess) Selected component [tm]\
[grsacc08:09178] ess:tm setting name\
[grsacc08:09178] ess:tm set name to [[58899,0],2]\
[grsacc09:19455] mca:base:select:( ess) Querying component [tm]\
[grsacc09:19455] mca:base:select:( ess) Query of component [tm] set priority to 30\
[grsacc09:19455] mca:base:select:( ess) Selected component [tm]\
[grsacc09:19455] ess:tm setting name\
[grsacc09:19455] ess:tm set name to [[58899,0],3]\
[grsacc10:31295] mca:base:select:( ess) Querying component [tm]\
[grsacc10:31295] mca:base:select:( ess) Query of component [tm] set priority to 30\
[grsacc10:31295] mca:base:select:( ess) Selected component [tm]\
[grsacc10:31295] ess:tm setting name\
[grsacc10:31295] ess:tm set name to [[58899,0],4]\
[grsacc09:19455] [[58899,0],3] Topology Info:\
[grsacc07:01601] [[58899,0],1] Topology Info:\
[grsacc08:09178] [[58899,0],2] Topology Info:\
[grsacc10:31295] [[58899,0],4] Topology Info:\
[grsacc09:19455] Type: Machine Number of child objects: 3\
Name=NULL\
total=25156656KB\
DMIProductName=X8DTT-H\
DMIProductVersion=1234567890\
DMIBoardVendor=Supermicro\
DMIBoardName=X8DTT-H\
DMIBoardVersion=1234567890\
DMIBoardAssetTag=1234567890\
DMIChass[grsacc07:01601] Type: Machine Number of child objects: 3\
Name=NULL\
total=25156656KB\
DMIProductName=X8DTT-H\
DMIProductVersion=1234567890\
DMIBoardVendor=Supermicro\
DMIBoardName=X8DTT-H\
DMIBoardVersion=1234567890\
DMIBoardAssetTag=1234567890\
DMIChassisVendor=Supermicro\
DMIChassisType=17\
DMIChassisVersion=1234567890\
DMIChassisAssetTag="To Be Filled By O.E.M."\
DMIBIOSVendor="American Megatrends Inc."\
DMIBIOSVersion="080015 "\
DMIBIOSDate=12/10/2009\
DMISysVendor=Supermicro\
Backend=Linux\
OSName=Li[grsacc10:31295] Type: Machine Number of child objects: 3\
Name=NULL\
total=25156656KB\
DMIProductName=X8DTT-H\
DMIProductVersion=1234567890\
DMIBoardVendor=Supermicro\
DMIBoardName=X8DTT-H\
DMIBoardVersion=1234567890\
DMIBoardAssetTag=1234567890\
DMIChassisVendor=Supermicro\
DMIChassisType=17\
DMIChassisVersion=1234567890\
DMIChassisAssetTag="To Be Filled By O.E.M."\
DMIBIOSVendor="American Megatrends Inc."\
DMIBIOSVersion="080015 "\
DMIBIOSDate=12/10/2009\
DMISysVendor=Supermicro\
Backend=Linux\
OSName=Li[grsacc08:09178] Type: Machine Number of child objects: 3\
Name=NULL\
total=25156656KB\
DMIProductName=X8DTT-H\
DMIProductVersion=1234567890\
DMIBoardVendor=Supermicro\
DMIBoardName=X8DTT-H\
DMIBoardVersion=1234567890\
DMIBoardAssetTag=1234567890\
DMIChassnux\
OSRelease=2.6.35-30-generic\
OSVersion="#56-Ubuntu SMP Mon Jul 11 20:01:08 UTC 2011"\
Architecture=x86_64\
Cpuset: 0x000000ff\
Online: 0x000000ff\
Allowed: 0x000000ff\
Bind CPU proc: TRUE\
Bind CPU thread: TRUE\
Bind MEM proc: FALSE\
Bind MEM thisVendor=Supermicro\
DMIChassisType=17\
DMIChassisVersion=1234567890\
DMIChassisAssetTag="To Be Filled By O.E.M."\
DMIBIOSVendor="American Megatrends Inc."\
DMIBIOSVersion="080015 "\
DMIBIOSDate=12/10/2009\
DMISysVendor=Supermicro\
Backend=Linux\
OSName=Linux\
OSRelease=2.6.35-30-generic\
OSVersion="#56-Ubuntu SMP Mon Jul 11 20:01:08 UTC 2011"\
Architecture=x86_64\
Cpuset: 0x000000ff\
Online: 0x000000ff\
Allowed: 0x000000ff\
Bind CPU proc: TRUE\
Bind CPU thread: TRUE\
Bind MEM proc: FALSE\
Bind MEM thisVendor=Supermicro\
DMIChassisType=17\
DMIChassisVersion=1234567890\
DMIChassisAssetTag="To Be Filled By O.E.M."\
DMIBIOSVendor="American Megatrends Inc."\
DMIBIOSVersion="080015 "\
DMIBIOSDate=12/10/2009\
DMISysVendor=Supermicro\
Backend=Linux\
OSName=Liread: TRUE\
Type: NUMANode Number of child objects: 1\
Name=NULL\
local=12573744KB\
total=12573744KB\
Cpuset: 0x0000000f\
Online: 0x0000000f\
Allowed: 0x0000000f\
Type: Socket Number of child objects: 1\
Name=NULL\
CPUModel="Intel(R) Xeon(R) Cnux\
OSRelease=2.6.35-30-generic\
OSVersion="#56-Ubuntu SMP Mon Jul 11 20:01:08 UTC 2011"\
Architecture=x86_64\
Cpuset: 0x000000ff\
Online: 0x000000ff\
Allowed: 0x000000ff\
Bind CPU proc: TRUE\
Bind CPU thread: TRUE\
Bind MEM proc: FALSE\
Bind MEM thread: TRUE\
Type: NUMANode Number of child objects: 1\
Name=NULL\
local=12573744KB\
total=12573744KB\
Cpuset: 0x0000000f\
Online: 0x0000000f\
Allowed: 0x0000000f\
Type: Socket Number of child objects: 1\
Name=NULL\
CPUModel="Intel(R) Xeon(R) Cnux\
OSRelease=2.6.35-30-generic\
OSVersion="#56-Ubuntu SMP Mon Jul 11 20:01:08 UTC 2011"\
Architecture=x86_64\
Cpuset: 0x000000ff\
Online: 0x000000ff\
Allowed: 0x000000ff\
Bind CPU proc: TRUE\
Bind CPU thread: TRUE\
Bind MEM proc: FALSE\
Bind MEM thPU X5570 @ 2.93GHz"\
Cpuset: 0x0000000f\
Online: 0x0000000f\
Allowed: 0x0000000f\
Type: L3Cache Number of child objects: 4\
Name=NULL\
size=8192KB\
linesize=64\
ways=16\
Cpuset: 0x0000000f\
Online: 0x0000000f\
Alread: TRUE\
Type: NUMANode Number of child objects: 1\
Name=NULL\
local=12573744KB\
total=12573744KB\
Cpuset: 0x0000000f\
Online: 0x0000000f\
Allowed: 0x0000000f\
Type: Socket Number of child objects: 1\
Name=NULL\
CPUModel="Intel(R) Xeon(R) CPU X5570 @ 2.93GHz"\
Cpuset: 0x0000000f\
Online: 0x0000000f\
Allowed: 0x0000000f\
Type: L3Cache Number of child objects: 4\
Name=NULL\
size=8192KB\
linesize=64\
ways=16\
Cpuset: 0x0000000f\
Online: 0x0000000f\
Alread: TRUE\
Type: NUMANode Number of child objects: 1\
Name=NULL\
local=12573744KB\
total=12573744KB\
Cpuset: 0x0000000f\
Online: 0x0000000f\
Allowed: 0x0000000f\
Type: Socket Number of child objects: 1\
Name=NULL\
CPUModel="Intel(R) Xeon(R) Clowed: 0x0000000f\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: L1dCache Number of child objects: 1\
Name=PU X5570 @ 2.93GHz"\
Cpuset: 0x0000000f\
Online: 0x0000000f\
Allowed: 0x0000000f\
Type: L3Cache Number of child objects: 4\
Name=NULL\
size=8192KB\
linesize=64\
ways=16\
Cpuset: 0x0000000f\
Online: 0x0000000f\
Allowed: 0x0000000f\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: L1dCache Number of child objects: 1\
Name=PU X5570 @ 2.93GHz"\
Cpuset: 0x0000000f\
Online: 0x0000000f\
Allowed: 0x0000000f\
Type: L3Cache Number of child objects: 4\
Name=NULL\
size=8192KB\
linesize=64\
ways=16\
Cpuset: 0x0000000f\
Online: 0x0000000f\
AlNULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000001\
Online: 0x00000001\
Allolowed: 0x0000000f\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000001\
Online: 0x00000001\
Allolowed: 0x0000000f\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: L1dCache Number of child objects: 1\
Name=wed: 0x00000001\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000001\
Online: 0x00000001\
Allo ways=8\
Cpuset: 0x00000002\
Online: 0x00000002\
Allowed: 0x00000002\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000002\
Online: 0x00000002\
Allwed: 0x00000001\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000002\
Online: 0x00000002\
Allowed: 0x00000002\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000002\
Online: 0x00000002\
Allwed: 0x00000001\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000001\
Online: 0x00000001\
Allowed: 0x00000001\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
owed: 0x00000002\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000002\
Online: 0x00000002\
Allowed: 0x00000002\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000002\
ways=8\
Cpuset: 0x00000002\
Online: 0x00000002\
Allowed: 0x00000002\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000002\
Online: 0x00000002\
Allowed: 0x00000002\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000002\
Online: 0x00000002\
Allowed: 0x00000002\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000002\
ways=8\
Cpuset: 0x00000002\
Online: 0x00000002\
Allowed: 0x00000002\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000002\
Online: 0x00000002\
AllOnline: 0x00000002\
Allowed: 0x00000002\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: L1dCache Number oowed: 0x00000002\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000002\
Online: 0x00000002\
Allowed: 0x00000002\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000002\
Online: 0x00000002\
Allowed: 0x00000002\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: L1dCache Number oowed: 0x00000002\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000002\
Online: 0x00000002\
Allowed: 0x00000002\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000002\
f child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000004\
OOnline: 0x00000002\
Allowed: 0x00000002\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000004\
OOnline: 0x00000002\
Allowed: 0x00000002\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: L1dCache Number online: 0x00000004\
Allowed: 0x00000004\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: L2Cache Number of child objects: 1\
Name=NULL\
f child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: L2Cache Number of child objects: 1\
Name=NULL\
f child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000004\
Osize=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000008\
nline: 0x00000004\
Allowed: 0x00000004\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000008\
nline: 0x00000004\
Allowed: 0x00000004\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000004\
Online: 0x00000004\
Allowed: 0x00000004\
Type: L2Cache Number of child objects: 1\
Name=NULL\
Online: 0x00000008\
Allowed: 0x00000008\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: PU Number of child objects: 0\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: PU Number of child objects: 0\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000008\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: NUMANode Number of child objects: 1\
Name=NULL\
local=12582912KB\
total=12582912KB\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: Socket Numb Online: 0x00000008\
Allowed: 0x00000008\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: NUMANode Number of child objects: 1\
Name=NULL\
local=12582912KB\
total=12582912KB\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: Socket Numb Online: 0x00000008\
Allowed: 0x00000008\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: PU Number of child objects: 0\
Name=NULL\
er of child objects: 1\
Name=NULL\
CPUModel="Intel(R) Xeon(R) CPU X5570 @ 2.93GHz"\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: L3Cache Number of child objects: 4\
Name=NULL\
size=8192KB\
linesize=64 Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: NUMANode Number of child objects: 1\
Name=NULL\
local=12582912KB\
total=12582912KB\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: Socket Number of child objects: 1\
Name=NULL\
CPUModel="Intel(R) Xeon(R) CPU X5570 @ 2.93GHz"\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: L3Cache Number of child objects: 4\
Name=NULL\
size=8192KB\
linesize=64 Cpuset: 0x00000008\
Online: 0x00000008\
Allowed: 0x00000008\
Type: NUMANode Number of child objects: 1\
Name=NULL\
local=12582912KB\
total=12582912KB\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: Socket Numb\
ways=16\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0xer of child objects: 1\
Name=NULL\
CPUModel="Intel(R) Xeon(R) CPU X5570 @ 2.93GHz"\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: L3Cache Number of child objects: 4\
Name=NULL\
size=8192KB\
linesize=64\
ways=16\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0xer of child objects: 1\
Name=NULL\
CPUModel="Intel(R) Xeon(R) CPU X5570 @ 2.93GHz"\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: L3Cache Number of child objects: 4\
Name=NULL\
size=8192KB\
linesize=6400000010\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: Core Number of child objects: 1\
Name=NUL\
ways=16\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: Core Number of child objects: 1\
Name=NUL\
ways=16\
Cpuset: 0x000000f0\
Online: 0x000000f0\
Allowed: 0x000000f0\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0xL\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: L2Cache Number of 00000010\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: L2Cache Number of 00000010\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: Core Number of child objects: 1\
Name=NULchild objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
wL\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
wL\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000010\
Online: 0x00000010\
Allowed: 0x00000010\
Type: L2Cache Number of ays=8\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: PU Number of chilchild objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: PU Number of chilchild objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
wd objects: 0\
Name=NULL\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000040\
Onlineays=8\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000040\
Onlineays=8\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: PU Number of chil: 0x00000040\
Allowed: 0x00000040\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x00000040\
Type: Core Number of chd objects: 0\
Name=NULL\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x00000040\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x00000040\
Type: Core Number of chd objects: 0\
Name=NULL\
Cpuset: 0x00000020\
Online: 0x00000020\
Allowed: 0x00000020\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000040\
Onlineild objects: 1\
Name=NULL\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x00000040\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x0000004: 0x00000040\
Allowed: 0x00000040\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x00000040\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x00000040\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x0000004: 0x00000040\
Allowed: 0x00000040\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x00000040\
Type: Core Number of ch0\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=ild objects: 1\
Name=NULL\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x00000040\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x00000040\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: L1dCache Number of child objects: [grsacc06:19901] [[58899,0],0] plm:base:orted_report_launch from daemon [[58899,0],1]\
[grsacc06:19901] [[58899,0],0] plm:base:orted_report_launch from daemon [[58899,0],1] on node grsacc07\
1\
Name=NULL\
size=ild objects: 1\
Name=NULL\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x00000040\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000040\
Online: 0x00000040\
Allowed: 0x000000432KB\
linesize=64\
ways=8\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
0\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000080\
[grsacc06:19901] [[58899,0],0] RECEIVED TOPOLOGY FROM NODE grsacc07\
Online: 0x00000080\
Allowed: 0x00000080\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size=32KB\
linesize=64\
ways=8\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
0\
Type: L2Cache Number of child objects: 1\
Name=NULL\
size=256KB\
linesize=64\
ways=8\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: L1dCache Number of child objects: 1\
Name=NULL\
size= Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: Bridge Host->PCI Number of child objects: 5\
Name=NULL\
buses=0000:[00-06]\
Type: PCI 8086:10d3 Numbe32KB\
linesize=64\
ways=8\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: Bridg[grsacc06:19901] [[58899,0],0] NEW TOPOLOGY - ADDING\
[grsacc06:19901] [[58899,0],0] plm:base:orted_report_launch completed for daemon [[58899,0],1] at contact 3860004864.1;tcp://192.168.222.7,192.168.111.7:51328\
e Host->PCI Number of child objects: 5\
Name=NULL\
buses=0000:[00-06]\
Type: PCI 8086:10d3 Numbe32KB\
linesize=64\
ways=8\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: Core Number of child objects: 1\
Name=NULL\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
r of child objects: 1\
Name=Intel Corporation 82574L Gigabit Network Connection\
busid=0000:01:00.0\
class=0200(Ether)\
PCIVendor="Intel Corporation"\
PCIDevice="82574L Gigabit Network Connection"\
Type: Network Number of child objects: 0\
N Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: Bridge Host->PCI Number of child objects: 5\
Name=NULL\
buses=0000:[00-06]\
Type: PCI 8086:10d3 Number of child objects: 1\
Name=Intel Corporation 82574L Gigabit Network Connection\
busid=0000:01:00.0\
class=0200(Ether)\
PCIVendor="Intel Corporation"\
PCIDevice="82574L Gigabit Network Connection"\
Type: Network Number of child objects: 0\
N Type: PU Number of child objects: 0\
Name=NULL\
Cpuset: 0x00000080\
Online: 0x00000080\
Allowed: 0x00000080\
Type: Bridge Host->PCI Number of child objects: 5\
Name=NULL\
buses=0000:[00-06]\
Type: PCI 8086:10d3 Numbeame=eth0\
Address=00:30:48:cf:64:ac\
Type: PCI 8086:10d3 Number of child objects: 1\
Name=Intel Corporation 82574L Gigabit Network Connection\
busid=0000:02:00.0\
class=0200(Ether)\
PCIVendor="Intel Corporation"\
PCIDevice="82574L Gigabit Netr of child objects: 1\
Name=Intel Corporation 82574L Gigabit Network Connection\
busid=0000:01:00.0\
class=0200(Ether)\
PCIVendor="Intel Corporation"\
PCIDevice="82574L Gigabit Network Connection"\
Type: Network Number of child objects: 0\
Name=eth0\
Address=00:30:48:cf:51:46\
Type: PCI 8086:10d3 Number of child objects: 1\
Name=Intel Corporation 82574L Gigabit Network Connection\
busid=0000:02:00.0\
class=0200(Ether)\
PCIVendor="Intel Corporation"\
PCIDevice="82574L Gigabit Netr of child objects: 1\
Name=Intel Corporation 82574L Gigabit Network Connection\
busid=0000:01:00.0\
class=0200(Ether)\
PCIVendor="Intel Corporation"\
PCIDevice="82574L Gigabit Network Connection"\
Type: Network Number of child objects: 0\
Nwork Connection"\
Type: Network Number of child objects: 0\
Name=eth1\
Address=00:30:48:cf:64:ad\
Type: PCI 15b3:673c Number of child objects: 2\
Name=Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]\
busid=0000:03ame=eth0\
Address=00:30:48:cf:5e:0a\
Type: PCI 8086:10d3 Number of child objects: 1\
Name=Intel Corporation 82574L Gigabit Network Connection\
busid=0000:02:00.0\
class=0200(Ether)\
PCIVendor="Intel Corporation"\
PCIDevice="82574L Gigabit Network Connection"\
Type: Network Number of child objects: 0\
Name=eth1\
Address=00:30:48:cf:51:47\
Type: PCI 15b3:673c Number of child objects: 2\
Name=Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]\
busid=0000:03ame=eth0\
Address=00:30:48:cf:63:9c\
Type: PCI 8086:10d3 Number of child objects: 1\
Name=Intel Corporation 82574L Gigabit Network Connection\
busid=0000:02:00.0\
class=0200(Ether)\
PCIVendor="Intel Corporation"\
PCIDevice="82574L Gigabit Net:00.0\
class=0c06(IB)\
PCIVendor="Mellanox Technologies"\
PCIDevice="MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]"\
Type: Network Number of child objects: 0\
Name=ib0\
Address=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:ce:1work Connection"\
Type: Network Number of child objects: 0\
Name=eth1\
Address=00:30:48:cf:5e:0b\
Type: PCI 15b3:673c Number of child objects: 2\
Name=Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]\
busid=0000:03:00.0\
class=0c06(IB)\
PCIVendor="Mellanox Technologies"\
PCIDevice="MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]"\
Type: Network Number of child objects: 0\
Name=ib0\
Address=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:ce:0work Connection"\
Type: Network Number of child objects: 0\
Name=eth1\
Address=00:30:48:cf:63:9d\
Type: PCI 15b3:673c Number of child objects: 2\
Name=Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]\
busid=0000:031:d5\
Port=1\
Type: OpenFabrics Number of child objects: 0\
Name=mlx4_0\
NodeGUID=0030:48ff:ffce:11d4\
SysImageGUID=0030:48ff:ffce:11d7\
Port1State=4\
Port1LID=0x9\
Port1LMC=0\
Port1GID0=fe80:0000:0000:0000:0030:48ff:ffce:11d5\
T:00.0\
class=0c06(IB)\
PCIVendor="Mellanox Technologies"\
PCIDevice="MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]"\
Type: Network Number of child objects: 0\
Name=ib0\
Address=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:ce:1f:95\
Port=1\
Type: OpenFabrics Number of child objects: 0\
Name=mlx4_0\
NodeGUID=0030:48ff:ffce:0f94\
SysImageGUID=0030:48ff:ffce:0f97\
Port1State=4\
Port1LID=0x6\
Port1LMC=0\
Port1GID0=fe80:0000:0000:0000:0030:48ff:ffce:0f95\
T:00.0\
class=0c06(IB)\
PCIVendor="Mellanox Technologies"\
PCIDevice="MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]"\
Type: Network Number of child objects: 0\
Name=ib0\
Address=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:ce:0ype: PCI 102b:0532 Number of child objects: 0\
Name=Matrox Graphics, Inc. MGA G200eW WPCM450\
busid=0000:06:01.0\
class=0300(VGA)\
PCIVendor="Matrox Graphics, Inc."\
PCIDevice="MGA G200eW WPCM450"\
Type: PCI 8086:3a20 Number of child objects: 1\
0:f5\
Port=1\
Type: OpenFabrics Number of child objects: 0\
Name=mlx4_0\
NodeGUID=0030:48ff:ffce:10f4\
SysImageGUID=0030:48ff:ffce:10f7\
Port1State=4\
Port1LID=0xa\
Port1LMC=0\
Port1GID0=fe80:0000:0000:0000:0030:48ff:ffce:10f5\
Type: PCI 102b:0532 Number of child objects: 0\
Name=Matrox Graphics, Inc. MGA G200eW WPCM450\
busid=0000:06:01.0\
class=0300(VGA)\
PCIVendor="Matrox Graphics, Inc."\
PCIDevice="MGA G200eW WPCM450"\
Type: PCI 8086:3a20 Number of child objects: 1\
f:d1\
Port=1\
Type: OpenFabrics Number of child objects: 0\
Name=mlx4_0\
NodeGUID=0030:48ff:ffce:0fd0\
SysImageGUID=0030:48ff:ffce:0fd3\
Port1State=4\
Port1LID=0x7\
Port1LMC=0\
Port1GID0=fe80:0000:0000:0000:0030:48ff:ffce:0fd1\
T Name=Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1\
busid=0000:00:1f.2\
class=0101(IDE)\
PCIVendor="Intel Corporation"\
PCIDevice="82801JI (ICH10 Family) 4 port SATA IDE Controller #1"\
Type: Block Number of child objeype: PCI 102b:0532 Number of child objects: 0\
Name=Matrox Graphics, Inc. MGA G200eW WPCM450\
busid=0000:06:01.0\
class=0300(VGA)\
PCIVendor="Matrox Graphics, Inc."\
PCIDevice="MGA G200eW WPCM450"\
Type: PCI 8086:3a20 Number of child objects: 1\
Name=Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1\
busid=0000:00:1f.2\
class=0101(IDE)\
PCIVendor="Intel Corporation"\
PCIDevice="82801JI (ICH10 Family) 4 port SATA IDE Controller #1"\
Type: Block Number of child objeype: PCI 102b:0532 Number of child objects: 0\
Name=Matrox Graphics, Inc. MGA G200eW WPCM450\
busid=0000:06:01.0\
class=0300(VGA)\
PCIVendor="Matrox Graphics, Inc."\
PCIDevice="MGA G200eW WPCM450"\
Type: PCI 8086:3a20 Number of child objects: 1\
cts: 0\
Name=sda\
[grsacc09:19455] mca:base:select:( plm) Querying component [rsh]\
[grsacc09:19455] [[58899,0],3] plm:rsh_lookup on agent ssh : rsh path NULL\
[grsacc09:19455] mca:base:select:( plm) Query of component [rsh] set priority to 10\
[grsacc09: Name=Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1\
busid=0000:00:1f.2\
class=0101(IDE)\
PCIVendor="Intel Corporation"\
PCIDevice="82801JI (ICH10 Family) 4 port SATA IDE Controller #1"\
Type: Block Number of child objects: 0\
Name=sda\
[grsacc07:01601] mca:base:select:( plm) Querying component [rsh]\
[grsacc07:01601] [[58899,0],1] plm:rsh_lookup on agent ssh : rsh path NULL\
[grsacc07:01601] mca:base:select:( plm) Query of component [rsh] set priority to 10\
[grsacc07: Name=Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1\
busid=0000:00:1f.2\
class=0101(IDE)\
PCIVendor="Intel Corporation"\
PCIDevice="82801JI (ICH10 Family) 4 port SATA IDE Controller #1"\
Type: Block Number of child obje19455] mca:base:select:( plm) Selected component [rsh]\
[grsacc09:19455] mca:base:select:(grpcomm) Querying component [bad]\
cts: 0\
Name=sda\
[grsacc10:31295] mca:base:select:( plm) Querying component [rsh]\
[grsacc10:31295] [[58899,0],4] plm:rsh_lookup on agent ssh : rsh path NULL\
[grsacc10:31295] mca:base:select:( plm) Query of component [rsh] set priority to 10\
[grsacc10:01601] mca:base:select:( plm) Selected component [rsh]\
[grsacc07:01601] mca:base:select:(grpcomm) Querying component [bad]\
cts: 0\
Name=sda\
[grsacc08:09178] mca:base:select:( plm) Querying component [rsh]\
[grsacc08:09178] [[58899,0],2] plm:rsh_lookup on agent ssh : rsh path NULL\
[grsacc08:09178] mca:base:select:( plm) Query of component [rsh] set priority to 10\
[grsacc08:[grsacc09:19455] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc09:19455] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc09:19455] [[58899,0],3] grpcomm:base:receive start comm\
[grsacc09:19455] [[58899,0],3] plm:rsh31295] mca:base:select:( plm) Selected component [rsh]\
[grsacc10:31295] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc07:01601] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc07:01601] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc07:01601] [[58899,0],1] grpcomm:base:receive start comm\
[grsacc07:01601] [[58899,0],1] plm:rsh09178] mca:base:select:( plm) Selected component [rsh]\
[grsacc08:09178] mca:base:select:(grpcomm) Querying component [bad]\
_setup on agent ssh : rsh path NULL\
[grsacc09:19455] [[58899,0],3] plm:base:receive start comm\
[grsacc09:19455] [[58899,0],3] setting up session dir with\
tmpdir: UNDEF\
host grsacc09\
[grsacc10:31295] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc10:31295] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc10:31295] [[58899,0],4] grpcomm:base:receive start comm\
[grsacc10:31295] [[58899,0],4] plm:rsh_setup on agent ssh : rsh path NULL\
[grsacc07:01601] [[58899,0],1] plm:base:receive start comm\
[grsacc07:01601] [[58899,0],1] setting up session dir with\
tmpdir: UNDEF\
host grsacc07\
[grsacc08:09178] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc08:09178] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc08:09178] [[58899,0],2] grpcomm:base:receive start comm\
[grsacc08:09178] [[58899,0],2] plm:rsh_setup on agent ssh : rsh path NULL\
[grsacc10:31295] [[58899,0],4] plm:base:receive start comm\
[grsacc10:31295] [[58899,0],4] setting up session dir with\
tmpdir: UNDEF\
host grsacc10\
_setup on agent ssh : rsh path NULL\
[grsacc08:09178] [[58899,0],2] plm:base:receive start comm\
[grsacc08:09178] [[58899,0],2] setting up session dir with\
tmpdir: UNDEF\
host grsacc08\
[grsacc06:19901] [[58899,0],0] plm:base:orted_report_launch from daemon [[58899,0],3]\
[grsacc06:19901] [[58899,0],0] plm:base:orted_report_launch from daemon [[58899,0],3] on node grsacc09\
[grsacc06:19901] [[58899,0],0] plm:base:orted_report_launch completed for daemon [[58899,0],3] at contact 3860004864.3;tcp://192.168.222.9,192.168.111.9:35816\
[grsacc06:19901] [[58899,0],0] plm:base:orted_report_launch from daemon [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] plm:base:orted_report_launch from daemon [[58899,0],2] on node grsacc08\
[grsacc06:19901] [[58899,0],0] plm:base:orted_report_launch completed for daemon [[58899,0],2] at contact 3860004864.2;tcp://192.168.222.8,192.168.111.8:52494\
[grsacc06:19901] [[58899,0],0] plm:base:orted_report_launch from daemon [[58899,0],4]\
[grsacc06:19901] [[58899,0],0] plm:base:orted_report_launch from daemon [[58899,0],4] on node grsacc10\
[grsacc06:19901] [[58899,0],0] plm:base:orted_report_launch completed for daemon [[58899,0],4] at contact 3860004864.4;tcp://192.168.222.10,192.168.111.10:48032\
[grsacc06:19901] [[58899,0],0] plm:base:launch_apps for job [58899,1]\
[grsacc06:19901] [[58899,0],0] grpcomm:bad:xcast sent to job [58899,0] tag 1\
[grsacc06:19901] [[58899,0],0] grpcomm:xcast:recv: with 1754 bytes\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],1]\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],4]\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 0\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 1\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 2\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] grpcomm:xcast:recv: with 1754 bytes\
[grsacc10:31295] [[58899,0],4] grpcomm:xcast:recv: with 1754 bytes\
[grsacc07:01601] [[58899,0],1] grpcomm:base:xcast updating daemon nidmap\
[grsacc08:09178] [[58899,0],2] grpcomm:xcast:recv: with 1754 bytes\
[grsacc07:01601] [[58899,0],1] orte:daemon:send_relay sending relay msg to [[58899,0],3]\
[grsacc08:09178] [[58899,0],2] grpcomm:base:xcast updating daemon nidmap\
[grsacc08:09178] [[58899,0],2] orte:daemon:send_relay - recipient list is empty!\
[grsacc10:31295] [[58899,0],4] grpcomm:base:xcast updating daemon nidmap\
[grsacc10:31295] [[58899,0],4] orte:daemon:send_relay - recipient list is empty!\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 0\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 0\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 0\
[grsacc07:01601] [[58899,0],1] COLLECTIVE 0 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc07:01601] [[58899,0],1] PR[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 0\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 0\
[grsacc08:09178] [[58899,0],2] COLLECTIVE 0 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc08:09178] [[58899,0],2] PR[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 0\
[grsacc10:31295] [[58899,0],4] COLLECTIVE 0 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 1\
[grsacc10:31295] [[58899,0],4] ALOGRESSING COLL id 1\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 0\
[grsacc07:01601] [[58899,0],1] COLLECTIVE 1 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 2\
[grsacc07:016OGRESSING COLL id 1\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 0\
[grsacc08:09178] [[58899,0],2] COLLECTIVE 1 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 2\
[grsacc08:091[grsacc09:19455] [[58899,0],3] grpcomm:xcast:recv: with 1754 bytes\
[grsacc09:19455] [[58899,0],3] grpcomm:base:xcast updating daemon nidmap\
[grsacc09:19455] [[58899,0],3] orte:daemon:send_relay - recipient list is empty!\
[grsacc09:19455] [[58899,0],3] PROGL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 0\
[grsacc10:31295] [[58899,0],4] COLLECTIVE 1 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 2\
[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,1] C01] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 0\
[grsacc07:01601] [[58899,0],1] COLLECTIVE 2 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],1]78] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 0\
[grsacc08:09178] [[58899,0],2] COLLECTIVE 2 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],2]RESSING COLL id 0\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 0\
[grsacc09:19455] [[58899,0],3] COLLECTIVE 0 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 1\
[grsacc09:19455ONTRIBUTE 0\
[grsacc10:31295] [[58899,0],4] COLLECTIVE 2 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],4]\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_c\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 3 ngr 0\
[grsacc07:0160\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],0]\
[grsacc08:] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 0\
[grsacc09:19455] [[58899,0],3] COLLECTIVE 1 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 2\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS Foll: WORKING COLLECTIVE 0\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],0]\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: MISSIN1] [[58899,0],1] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],1]\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: WORKING COLLECTIVE 1\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc07:01601] [[09178] [[58899,0],2] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 3 ngr 0\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],2]\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: WORKING COLLECTIVE OR JOB [58899,1] CONTRIBUTE 0\
[grsacc09:19455] [[58899,0],3] COLLECTIVE 2 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],3]\
[grsacc09:19455] [[58899,0],3] grpG CONTRIBUTORS: np 3 ngr 0\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],4[grsacc06:19901] [[58899,0],0] plm:base:launch wiring up iof for job [58899,1]\
]\
[g[pid 19904] starting up!\
[pid 19905] starting up!\
rsacc1[grsacc06:19901] [[58899,0],0] plm:base:receive processing msg\
[grsacc06:19901] [[58899,0],0] plm:base:receive update proc state command from [[58899,0],3]\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for job [58899,1]\
[grsacc06:19901] [[58899,0],0] plm:base:receive done processing commands\
[grsacc06:19904] mca:base:select:( ess) Querying component [env]\
[grsacc06:19904] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc06:19904] mca:base:select:( ess) Selected component [env]\
[grsacc06:19904] ess:env set name to [[58899,1],0]\
0:31295] [[58899,0],4] grpcomm:base:daemon_coll: WORKING COLLECTIVE 1\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_58899,0],1] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 3 ngr 0\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],1]\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: WORKING COLLECTIVE 2\
[grsacc1\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],0]\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 3 ngrcomm:base:daemon_coll: WORKING COLLECTIVE 0\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],1]\
[grsacc09:19455] [[58899,0],3] grpcomm:base:dcoll: NUM CONTRIBS: 0\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],0]\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 3 ngr [grsacc06:19905] mca:base:select:( ess) Querying component [env]\
[grsacc06:19905] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc06:19905] mca:base:select:( ess) Selected component [env]\
[grsacc06:19905] ess:env set name to [[58899,1],1]\
0\
[grsacc10:31295] [[588[pid 19906] starting up!\
99,0],4] grpcomm:base:daemon07:01601] [[58899,0],1] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 3 ngr 0\
0\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],2]\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: WORKING COLLECTIVE 2\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[gaemon_coll: MISSING CONTRIBUTORS: np 3 ngr 0\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],3]\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: WORKING COLLECTIVE 1\
[grsacc09:19455] [[58899,0],3] gr_coll: daemon collective recvd from [[58899,0],4]\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: WORKING COLLECTIVE 2\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_collrsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],0]\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 3 ngr 0\
pcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],1]\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 3 ngr 0\
[grsacc09:19455] [[58899,0],3] g: RELAYING COLLECTIVE TO [[58899,0],0]\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 3 ngr 0\
rpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],3]\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: WORKING COLLECTIVE 2\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],1]\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 3 ngr 0\
[grsacc06:19906] mca:base:select:( ess) Querying component [env]\
[grsacc06:19906] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc06:19906] mca:base:select:( ess) Selected component [env]\
[grsacc06:19906] ess:env set name to [[58899,1],2]\
[grsacc06:19904] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc06:19904] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc06:19904] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive start comm\
[grsacc06:19904] [[58899,1],0] setting up session dir with\
tmpdir: /tmp\
host grsacc06\
[grsacc06:19905] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc06:19905] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc06:19905] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive start comm\
[grsacc06:19905] [[58899,1],1] setting up session dir with\
tmpdir: /tmp\
host grsacc06\
[grsacc06:19906] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc06:19906] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc06:19906] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive start comm\
[grsacc06:19906] [[58899,1],2] setting up session dir with\
tmpdir: /tmp\
host grsacc06\
[grsacc06:19901] [[58899,0],0] plm:base:receive processing msg\
[grsacc06:19901] [[58899,0],0] plm:base:receive update proc state command from [[58899,0],1]\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for job [58899,1]\
[grsacc06:19901] [[58899,0],0] plm:base:receive done processing commands\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: CANNOT PROGRESS\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],4]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: CANNOT PROGRESS\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 1\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: CANNOT PROGRESS\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],4]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 1\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: CANNOT PROGRESS\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 2\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: CANNOT PROGRESS\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],4]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 2\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: CANNOT PROGRESS\
[grsacc06:19901] [[58899,0],0] plm:base:receive processing msg\
[grsacc06:19901] [[58899,0],0] plm:base:receive update proc state command from [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for job [58899,1]\
[grsacc06:19901] [[58899,0],0] plm:base:receive done processing commands\
[grsacc06:19901] [[58899,0],0] plm:base:receive processing msg\
[grsacc06:19901] [[58899,0],0] plm:base:receive update proc state command from [[58899,0],4]\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for job [58899,1]\
[grsacc06:19901] [[58899,0],0] plm:base:receive done processing commands\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],3]\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: WORKI[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],1]\
NG C[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0\
OLL[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: CANNOT PROGRESS\
ECTIVE 0\
[grsacc07:01601] [[58899,0],1] grp[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],1]\
com[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 1\
m:b[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: CANNOT PROGRESS\
ase:daemon_coll: NUM CONTRIBS: 0\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: RELAYING COLLECT[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],1]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 2\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: CANNOT PROGRESS\
IVE TO [[58899,0],0]\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 3 ngr 0\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],3]\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: WORKING COLLECTIVE 1\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],0]\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 3 ngr 0\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],3]\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: WORKING COLLECTIVE 2\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],0]\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 3 ngr 0\
[grsacc06:19901] [[58899,0],0] plm:base:launch [58899,1] registered\
[grsacc06:19901] [[58899,0],0] plm:base:launch job [58899,1] is not a dynamic spawn\
[grsacc06:19904] [[58899,1],0] Not bound at launch\
[grsacc06:19906] [[58899,1],2] Not bound at launch\
[grsacc06:19905] [[58899,1],1] Not bound at launch\
[grsacc06:19906] [[58899,1],2] ACTIVATING GRCPCOMM OP 0 at grpcomm_bad_module.c:321\
[grsacc06:19906] [[58899,1],2] grpcomm:base:modex: performing modex\
[grsacc06:19906] [[58899,1],2] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc06:19906] [[58899,1],2] grpcomm:base:full:modex: executing allgather\
[grsacc06:19906] [[58899,1],2] grpcomm:bad entering allgather\
[grsacc06:19906] [[58899,1],2] ACTIVATING GRCPCOMM OP 0 at grpcomm_bad_module.c:311\
[grsacc06:19906] [[58899,1],2] grpcomm:base:modex: modex posted\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 0 to our daemon\
[grsacc06:19906] [[58899,1],2] grpcomm:bad allgather underway\
[grsacc06:19904] [[58899,1],0] ACTIVATING GRCPCOMM OP 0 at grpcomm_bad_module.c:321\
[grsacc06:19904] [[58899,1],0] grpcomm:base:modex: performing modex\
[grsacc06:19904] [[58899,1],0] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc06:19904] [[58899,1],0] grpcomm:base:full:modex: executing allgather\
[grsacc06:19904] [[58899,1],0] grpcomm:bad entering allgather\
[grsacc06:19904] [[58899,1],0] ACTIVATING GRCPCOMM OP 0 at grpcomm_bad_module.c:311\
[grsacc06:19904] [[58899,1],0] grpcomm:base:modex: modex posted\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 0 to our daemon\
[grsacc06:19904] [[58899,1],0] grpcomm:bad allgather underway\
[grsacc06:19905] [[58899,1],1] ACTIVATING GRCPCOMM OP 0 at grpcomm_bad_module.c:321\
[grsacc06:19905] [[58899,1],1] grpcomm:base:modex: performing modex\
[grsacc06:19905] [[58899,1],1] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc06:19905] [[58899,1],1] grpcomm:base:full:modex: executing allgather\
[grsacc06:19905] [[58899,1],1] grpcomm:bad entering allgather\
[grsacc06:19905] [[58899,1],1] ACTIVATING GRCPCOMM OP 0 at grpcomm_bad_module.c:311\
[grsacc06:19905] [[58899,1],1] grpcomm:base:modex: modex posted\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 0 to our daemon\
[grsacc06:19905] [[58899,1],1] grpcomm:bad allgather underway\
[grsacc06:19901] [[58899,0],0] COLLECTIVE RECVD FROM [[58899,1],1]\
[grsacc06:19901] [[58899,0],0] WORKING COLLECTIVE 0\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLLECTIVE 0\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 0\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 1\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 2\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] COLLECTIVE RECVD FROM [[58899,1],2]\
[grsacc06:19901] [[58899,0],0] WORKING COLLECTIVE 0\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLLECTIVE 0\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 0\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 1\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 2\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] COLLECTIVE RECVD FROM [[58899,1],0]\
[grsacc06:19901] [[58899,0],0] WORKING COLLECTIVE 0\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLLECTIVE 0\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 0\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] COLLECTIVE 0 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 1\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 2\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],0]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc06:19901] [[58899,0],0] grpcomm:bad:xcast sent to job [58899,1] tag 30\
[grsacc06:19901] [[58899,0],0] grpcomm:xcast:recv: with 460 bytes\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],1]\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],4]\
[grsacc08:09178] [[58899,0],2] grpcomm:xcast:recv: with 460 bytes\
[grsacc08:09178] [[58899,0],2] orte:daemon:send_relay - recipient list is empty!\
[grsacc07:01601] [[58899,0],1] grpcomm:xcast:recv: with 460 bytes\
[grsacc07:01601] [[58899,0],1] orte:daemon:send_relay sending relay msg to [[58899,0],3]\
[grsacc10:31295] [[58899,0],4] grpcomm:xcast:recv: with 460 bytes\
[grsacc10:31295] [[58899,0],4] orte:daemon:send_relay - recipient list is empty!\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 0 recvd from [[58899,0],0]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 0\
[grsacc06:19904] [[58899,1],0] STORING MODEX DATA\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,1],1]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,1],2]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,1],0]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc06:19904] [[58899,1],0] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 0 recvd from [[58899,0],0]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 0\
[grsacc06:19906] [[58899,1],2] STORING MODEX DATA\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,1],1]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,1],2]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,1],0]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc06:19906] [[58899,1],2] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 0 recvd from [[58899,0],0]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 0\
[grsacc06:19905] [[58899,1],1] STORING MODEX DATA\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,1],1]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,1],2]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,1],0]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc06:19905] [[58899,1],1] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc06:19906] [[58899,1],2] grpcomm:bad entering barrier\
[grsacc06:19906] [[58899,1],2] ACTIVATING GRCPCOMM OP 1 at grpcomm_bad_module.c:206\
[grsacc06:19904] [[58899,1],0] grpcomm:bad entering barrier\
[grsacc06:19904] [[58899,1],0] ACTIVATING GRCPCOMM OP 1 at grpcomm_bad_module.c:206\
[grsacc06:19905] [[58899,1],1] grpcomm:bad entering barrier\
[grsacc06:19905] [[58899,1],1] ACTIVATING GRCPCOMM OP 1 at grpcomm_bad_module.c:206\
[grsacc06:19904] [[58899,1],0] grpcomm:bad barrier underway\
[grsacc06:19906] [[58899,1],2] grpcomm:bad barrier underway\
[grsacc06:19905] [[58899,1],1] grpcomm:bad barrier underway\
[grsacc09:19455] [[58899,0],3] grpcomm:xcast:recv: with 460 bytes\
[grsacc09:19455] [[58899,0],3] orte:daemon:send_relay - recipient list is empty!\
[grsacc06:19901] [[58899,0],0] COLLECTIVE RECVD FROM [[58899,1],1]\
[grsacc06:19901] [[58899,0],0] WORKING COLLECTIVE 1\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLLECTIVE 1\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 1\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 2\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] COLLECTIVE RECVD FROM [[58899,1],2]\
[grsacc06:19901] [[58899,0],0] WORKING COLLECTIVE 1\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLLECTIVE 1\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 1\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 2\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] COLLECTIVE RECVD FROM [[58899,1],0]\
[grsacc06:19901] [[58899,0],0] WORKING COLLECTIVE 1\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLLECTIVE 1\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 1\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] COLLECTIVE 1 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 2\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],0]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 1\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc06:19901] [[58899,0],0] grpcomm:bad:xcast sent to job [58899,1] tag 30\
[grsacc06:19901] [[58899,0],0] grpcomm:xcast:recv: with 40 bytes\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],1]\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],4]\
[grsacc08:09178] [[58899,0],2] grpcomm:xcast:recv: with 40 bytes\
[grsacc10:31295] [[58899,0],4] grpcomm:xcast:recv: with 40 bytes\
[grsacc07:01601] [[58899,0],1] grpcomm:xcast:recv: with 40 bytes\
[grsacc08:09178] [[58899,0],2] orte:daemon:send_relay - recipient list is empty!\
[grsacc10:31295] [[58899,0],4] orte:daemon:send_relay - recipient list is empty!\
[grsacc07:01601] [[58899,0],1] orte:daemon:send_relay sending relay msg to [[58899,0],3]\
[grsacc09:19455] [[58899,0],3] grpcomm:xcast:recv: with 40 bytes\
[grsacc09:19455] [[58899,0],3] orte:daemon:send_relay - recipient list is empty!\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 1 recvd from [[58899,0],0]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 1\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 1 recvd from [[58899,0],0]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 1\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 1 recvd from [[58899,0],0]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 1\
1 completed MPI_Init\
Parent [pid 19905] about to spawn!\
0 completed MPI_Init\
Parent [pid 19904] about to spawn!\
2 completed MPI_Init\
Parent [pid 19906] about to spawn!\
[grsacc06:19901] [[58899,0],0] plm:base:receive processing msg\
[grsacc06:19901] [[58899,0],0] plm:base:receive job launch command from [[58899,1],0]\
[grsacc06:19901] [[58899,0],0] plm:base:receive adding hosts\
[grsacc06:19901] [[58899,0],0] plm:base:receive calling spawn\
[grsacc06:19901] [[58899,0],0] plm:base:receive done processing commands\
[grsacc06:19901] [[58899,0],0] plm:base:setup_job\
[grsacc06:19901] [[58899,0],0] plm:base:setup_vm\
[grsacc06:19901] [[58899,0],0] plm:base:launch_apps for job [58899,2]\
[grsacc06:19901] [[58899,0],0] grpcomm:bad:xcast sent to job [58899,0] tag 1\
[grsacc06:19901] [[58899,0],0] grpcomm:xcast:recv: with 3987 bytes\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],1]\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],4]\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 2\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,1] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 3\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 0\
[grsacc06:19901] [[58899,0],0] COLLECTIVE 3 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 4\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 0\
[grsacc06:19901] [[58899,0],0] COLLECTIVE 4 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc06:19901] [[58899,0],0] PROGRESSING COLL id 5\
[grsacc06:19901] [[58899,0],0] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 0\
[grsacc06:19901] [[58899,0],0] COLLECTIVE 5 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],0]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 3\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],0]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 4\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],0]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 5\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 0\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 0\
[grsacc08:09178] [[58899,0],2] grpcomm:xcast:recv: with 3987 bytes\
[grsacc08:09178] [[58899,0],2] grpcomm:base:xcast updating daemon nidmap\
[grsacc10:31295] [[58899,0],4] grpcomm:xcast:recv: with 3987 bytes\
[grsacc07:01601] [[58899,0],1] grpcomm:xcast:recv: with 3987 bytes\
[grsacc10:31295] [[58899,0],4] grpcomm:base:xcast updating daemon nidmap\
[grsacc07:01601] [[58899,0],1] grpcomm:base:xcast updating daemon nidmap\
[grsacc08:09178] [[58899,0],2] orte:daemon:send_relay - recipient list is empty!\
[grsacc07:01601] [[58899,0],1] orte:daemon:send_relay sending relay msg to [[58899,0],3]\
[grsacc10:31295] [[58899,0],4] orte:daemon:send_relay - recipient list is empty!\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 0\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 0\
[grsacc08:09178] [[58899,0],2] COLL 0 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 1\
[grsacc08:09178] [[58899,0],2] COLL 1 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 2\
[grsacc08:09178] [[58899,0],2] COL[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 0\
L 2 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 3\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 4\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS F[grsacc10:31295] [[58899,0],4] COLL 0 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 1\
[grsacc10:31295] [[58899,0],4] COLL 1 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 2\
[grsacc10:31295] [[58899,0],4] COL[grsacc07:01601] [[58899,0],1] COLL 0 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 1\
[grsacc07:01601] [[58899,0],1] COLL 1 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 2\
[grsacc07:01601] [[58899,0],1] COLOR JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 5\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3] grpcomm:xcast:recv: with 3987 bytes\
[grsacc09:19455] [[58899,0],3] grpcomm:base:xcast updating daemon nidmap\
[grsacc09:19455] [[58899,0],3] orte:daemon:send_relay - recipient list is empty!\
L 2 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 3\
[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 4\
[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FL 2 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 3\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 4\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 5\
[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
OR JOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 5\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 0\
[grsacc09:19455] [[58899,0],3] COLL 0 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 1\
[grsacc09:19455] [[58899,0],3] COLL 1 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 2\
[grsacc09:19455] [[58899,0],3] COLL 2 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 3\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 4\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 5\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc06:19901] [[58899,0],0] plm:base:receive processing msg\
[grsacc06:19901] [[58899,0],0] plm:base:receive update proc state command from [[58899,0],4]\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for job [58899,2]\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for vpid 9 state RUNNING exit_code 0\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for vpid 10 state RUNNING exit_code 0\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for vpid 11 state RUNNING exit_code 0\
[grsacc06:19901] [[58899,0],0] plm:base:receive done processing commands\
[grsacc06:19901] [[58899,0],0] plm:base:receive processing msg\
[grsacc06:19901] [[58899,0],0] plm:base:receive update proc state command from [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for job [58899,2]\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for vpid 3 state RUNNING exit_code 0\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for vpid 4 state RUNNING exit_code 0\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for vpid 5 state RUNNING exit_code 0\
[grsacc06:19901] [[58899,0],0] plm:base:receive done processing commands\
[grsacc06:19901] [[58899,0],0] plm:base:receive processing msg\
[grsacc06:19901] [[58899,0],0] plm:base:receive update proc state command from [[58899,0],3]\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for job [58899,2]\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for vpid 6 state RUNNING exit_code 0\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for vpid 7 state RUNNING exit_code 0\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for vpid 8 state RUNNING exit_code 0\
[grsacc06:19901] [[58899,0],0] plm:base:receive done processing commands\
[grsacc06:19901] [[58899,0],0] plm:base:receive processing msg\
[grsacc06:19901] [[58899,0],0] plm:base:receive update proc state command from [[58899,0],1]\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for job [58899,2]\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for vpid 0 state RUNNING exit_code 0\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for vpid 1 state RUNNING exit_code 0\
[grsacc06:19901] [[58899,0],0] plm:base:receive got update_proc_state for vpid 2 state RUNNING exit_code 0\
[grsacc06:19901] [[58899,0],0] plm:base:receive done processing commands\
[grsacc06:19901] [[58899,0],0] plm:base:launch wiring up iof for job [58899,2]\
[pid 31298] starting up!\
[pid 31296] starting up!\
[pid 31297] starting up!\
[pid 19456] starting up!\
[pid 19457] starting up!\
[pid 19458] starting up!\
[pid 9181] starting up!\
[pid 9179] starting up!\
[pid 9180] starting up!\
[pid 1602] starting up!\
[pid 1603] starting up!\
[pid 1604] starting up!\
[grsacc10:31297] mca:base:select:( ess) Querying component [env]\
[grsacc10:31297] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc10:31297] mca:base:select:( ess) Selected component [env]\
[grsacc10:31297] ess:env set name to [[58899,2],10]\
[grsacc08:09181] mca:base:select:( ess) Querying component [env]\
[grsacc08:09181] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc08:09181] mca:base:select:( ess) Selected component [env]\
[grsacc08:09181] ess:env set name to [[58899,2],5]\
[grsacc10:31296] mca:base:select:( ess) Querying component [env]\
[grsacc10:31296] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc10:31296] mca:base:select:( ess) Selected component [env]\
[grsacc10:31298] mca:base:select:( ess) Querying component [env]\
[grsacc10:31298] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc10:31298] mca:base:select:( ess) Selected component [env]\
[grsacc10:31296] ess:env set name to [[58899,2],9]\
[grsacc10:31298] ess:env set name to [[58899,2],11]\
[grsacc07:01602] mca:base:select:( ess) Querying component [env]\
[grsacc07:01602] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc07:01602] mca:base:select:( ess) Selected component [env]\
[grsacc08:09179] mca:base:select:( ess) Querying component [env]\
[grsacc08:09179] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc08:09179] mca:base:select:( ess) Selected component [env]\
[grsacc07:01603] mca:base:select:( ess) Querying component [env]\
[grsacc07:01603] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc07:01603] mca:base:select:( ess) Selected component [env]\
[grsacc08:09179] ess:env set name to [[58899,2],3]\
[grsacc07:01602] ess:env set name to [[58899,2],0]\
[grsacc07:01603] ess:env set name to [[58899,2],1]\
[grsacc08:09180] mca:base:select:( ess) Querying component [env]\
[grsacc08:09180] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc08:09180] mca:base:select:( ess) Selected component [env]\
[grsacc07:01604] mca:base:select:( ess) Querying component [env]\
[grsacc07:01604] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc07:01604] mca:base:select:( ess) Selected component [env]\
[grsacc07:01604] ess:env set name to [[58899,2],2]\
[grsacc08:09180] ess:env set name to [[58899,2],4]\
[grsacc09:19457] mca:base:select:( ess) Querying component [env]\
[grsacc09:19457] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc09:19457] mca:base:select:( ess) Selected component [env]\
[grsacc09:19456] mca:base:select:( ess) Querying component [env]\
[grsacc09:19456] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc09:19456] mca:base:select:( ess) Selected component [env]\
[grsacc09:19457] ess:env set name to [[58899,2],7]\
[grsacc09:19456] ess:env set name to [[58899,2],6]\
[grsacc09:19458] mca:base:select:( ess) Querying component [env]\
[grsacc09:19458] mca:base:select:( ess) Query of component [env] set priority to 20\
[grsacc09:19458] mca:base:select:( ess) Selected component [env]\
[grsacc09:19458] ess:env set name to [[58899,2],8]\
[grsacc07:01604] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc07:01604] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc07:01604] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive start comm\
[grsacc07:01604] [[58899,2],2] setting up session dir with\
tmpdir: /tmp\
host grsacc07\
[grsacc10:31297] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc10:31297] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc10:31297] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive start comm\
[grsacc10:31297] [[58899,2],10] setting up session dir with\
tmpdir: /tmp\
host grsacc10\
[grsacc10:31298] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc10:31298] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc10:31298] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive start comm\
[grsacc10:31298] [[58899,2],11] setting up session dir with\
tmpdir: /tmp\
host grsacc10\
[grsacc08:09179] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc08:09179] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc08:09179] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive start comm\
[grsacc08:09179] [[58899,2],3] setting up session dir with\
tmpdir: /tmp\
host grsacc08\
[grsacc07:01602] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc07:01602] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc07:01602] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive start comm\
[grsacc07:01602] [[58899,2],0] setting up session dir with\
tmpdir: /tmp\
host grsacc07\
[grsacc08:09181] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc08:09181] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc08:09181] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc08:09180] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc08:09180] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc08:09180] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive start comm\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive start comm\
[grsacc08:09181] [[58899,2],5] setting up session dir with\
tmpdir: /tmp\
host grsacc08\
[grsacc08:09180] [[58899,2],4] setting up session dir with\
tmpdir: /tmp\
host grsacc08\
[grsacc09:19456] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc09:19456] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc09:19456] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive start comm\
[grsacc09:19456] [[58899,2],6] setting up session dir with\
tmpdir: /tmp\
host grsacc09\
[grsacc09:19458] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc09:19458] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc09:19458] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive start comm\
[grsacc09:19458] [[58899,2],8] setting up session dir with\
tmpdir: /tmp\
host grsacc09\
[grsacc10:31296] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc10:31296] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc10:31296] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive start comm\
[grsacc10:31296] [[58899,2],9] setting up session dir with\
tmpdir: /tmp\
host grsacc10\
[grsacc09:19457] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc09:19457] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc09:19457] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive start comm\
[grsacc09:19457] [[58899,2],7] setting up session dir with\
tmpdir: /tmp\
host grsacc09\
[grsacc07:01603] mca:base:select:(grpcomm) Querying component [bad]\
[grsacc07:01603] mca:base:select:(grpcomm) Query of component [bad] set priority to 10\
[grsacc07:01603] mca:base:select:(grpcomm) Selected component [bad]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive start comm\
[grsacc07:01603] [[58899,2],1] setting up session dir with\
tmpdir: /tmp\
host grsacc07\
[grsacc06:19901] [[58899,0],0] plm:base:receive processing msg\
[grsacc06:19901] [[58899,0],0] plm:base:receive done processing commands\
[grsacc06:19901] [[58899,0],0] plm:base:receive processing msg\
[grsacc06:19901] [[58899,0],0] plm:base:receive done processing commands\
[grsacc09:19458] [[58899,2],8] Not bound at launch\
[grsacc10:31298] [[58899,2],11] Not bound at launch\
[grsacc06:19901] [[58899,0],0] plm:base:receive processing msg\
[grsacc06:19901] [[58899,0],0] plm:base:receive done processing commands\
[grsacc09:19457] [[58899,2],7] Not bound at launch\
[grsacc10:31297] [[58899,2],10] Not bound at launch\
[grsacc10:31296] [[58899,2],9] Not bound at launch\
[grsacc07:01602] [[58899,2],0] Not bound at launch\
[grsacc08:09179] [[58899,2],3] Not bound at launch\
[grsacc07:01603] [[58899,2],1] Not bound at launch\
[grsacc08:09180] [[58899,2],4] Not bound at launch\
[grsacc08:09181] [[58899,2],5] Not bound at launch\
[grsacc07:01604] [[58899,2],2] Not bound at launch\
[grsacc06:19901] [[58899,0],0] plm:base:receive processing msg\
[grsacc06:19901] [[58899,0],0] plm:base:receive done processing commands\
[grsacc06:19901] [[58899,0],0] plm:base:launch [58899,2] registered\
[grsacc06:19901] [[58899,0],0] plm:base:launch sending dyn release of job [58899,2] to [[58899,1],0]\
[grsacc09:19456] [[58899,2],6] Not bound at launch\
[grsacc09:19457] [[58899,2],7] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:321\
[grsacc09:19458] [[58899,2],8] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:321\
[grsacc09:19457] [[58899,2],7] grpcomm:base:modex: performing modex\
[grsacc09:19457] [[58899,2],7] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc09:19457] [[58899,2],7] grpcomm:base:full:modex: executing allgather\
[grsacc09:19457] [[58899,2],7] grpcomm:bad entering allgather\
[grsacc09:19457] [[58899,2],7] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:311\
[grsacc09:19457] [[58899,2],7] grpcomm:base:modex: modex posted\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 3 to our daemon\
[grsacc09:19457] [[58899,2],7] grpcomm:bad allgather underway\
[grsacc09:19458] [[58899,2],8] grpcomm:base:modex: performing modex\
[grsacc09:19458] [[58899,2],8] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc09:19458] [[58899,2],8] grpcomm:base:full:modex: executing allgather\
[grsacc09:19458] [[58899,2],8] grpcomm:bad entering allgather\
[grsacc09:19458] [[58899,2],8] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:311\
[grsacc09:19458] [[58899,2],8] grpcomm:base:modex: modex posted\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 3 to our daemon\
[grsacc09:19458] [[58899,2],8] grpcomm:bad allgather underway\
[grsacc10:31298] [[58899,2],11] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:321\
[grsacc10:31297] [[58899,2],10] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:321\
[grsacc10:31297] [[58899,2],10] grpcomm:base:modex: performing modex\
[grsacc10:31297] [[58899,2],10] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc10:31297] [[58899,2],10] grpcomm:base:full:modex: executing allgather\
[grsacc10:31297] [[58899,2],10] grpcomm:bad entering allgather\
[grsacc10:31297] [[58899,2],10] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:311\
[grsacc10:31296] [[58899,2],9] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:321\
[grsacc10:31296] [[58899,2],9] grpcomm:base:modex: performing modex\
[grsacc10:31296] [[58899,2],9] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc10:31296] [[58899,2],9] grpcomm:base:full:modex: executing allgather\
[grsacc10:31296] [[58899,2],9] grpcomm:bad entering allgather\
[grsacc10:31296] [[58899,2],9] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:311\
[grsacc10:31296] [[58899,2],9] grpcomm:base:modex: modex posted\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 3 to our daemon\
[grsacc10:31298] [[58899,2],11] grpcomm:base:modex: performing modex\
[grsacc10:31298] [[58899,2],11] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc10:31298] [[58899,2],11] grpcomm:base:full:modex: executing allgather\
[grsacc10:31298] [[58899,2],11] grpcomm:bad entering allgather\
[grsacc10:31298] [[58899,2],11] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:311\
[grsacc10:31298] [[58899,2],11] grpcomm:base:modex: modex posted\
[grsacc10:31296] [[58899,2],9] grpcomm:bad allgather underway\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 3 to our daemon\
[grsacc10:31298] [[58899,2],11] grpcomm:bad allgather underway\
[grsacc10:31297] [[58899,2],10] grpcomm:base:modex: modex posted\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 3 to our daemon\
[grsacc10:31297] [[58899,2],10] grpcomm:bad allgather underway\
[grsacc08:09180] [[58899,2],4] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:321\
[grsacc08:09181] [[58899,2],5] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:321\
[grsacc08:09181] [[58899,2],5] grpcomm:base:modex: performing modex\
[grsacc08:09180] [[58899,2],4] grpcomm:base:modex: performing modex\
[grsacc08:09180] [[58899,2],4] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc08:09180] [[58899,2],4] grpcomm:base:full:modex: executing allgather\
[grsacc08:09180] [[58899,2],4] grpcomm:bad entering allgather\
[grsacc08:09180] [[58899,2],4] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:311\
[grsacc08:09180] [[58899,2],4] grpcomm:base:modex: modex posted\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 3 to our daemon\
[grsacc08:09180] [[58899,2],4] grpcomm:bad allgather underway\
[grsacc08:09181] [[58899,2],5] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc08:09181] [[58899,2],5] grpcomm:base:full:modex: executing allgather\
[grsacc08:09181] [[58899,2],5] grpcomm:bad entering allgather\
[grsacc08:09181] [[58899,2],5] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:311\
[grsacc08:09181] [[58899,2],5] grpcomm:base:modex: modex posted\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 3 to our daemon\
[grsacc08:09181] [[58899,2],5] grpcomm:bad allgather underway\
[grsacc08:09179] [[58899,2],3] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:321\
[grsacc08:09179] [[58899,2],3] grpcomm:base:modex: performing modex\
[grsacc08:09179] [[58899,2],3] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc08:09179] [[58899,2],3] grpcomm:base:full:modex: executing allgather\
[grsacc08:09179] [[58899,2],3] grpcomm:bad entering allgather\
[grsacc08:09179] [[58899,2],3] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:311\
[grsacc08:09179] [[58899,2],3] grpcomm:base:modex: modex posted\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 3 to our daemon\
[grsacc08:09179] [[58899,2],3] grpcomm:bad allgather underway\
[grsacc07:01603] [[58899,2],1] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:321\
[grsacc07:01603] [[58899,2],1] grpcomm:base:modex: performing modex\
[grsacc07:01603] [[58899,2],1] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc07:01603] [[58899,2],1] grpcomm:base:full:modex: executing allgather\
[grsacc07:01603] [[58899,2],1] grpcomm:bad entering allgather\
[grsacc07:01603] [[58899,2],1] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:311\
[grsacc07:01603] [[58899,2],1] grpcomm:base:modex: modex posted\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 3 to our daemon\
[grsacc07:01603] [[58899,2],1] grpcomm:bad allgather underway\
[grsacc07:01602] [[58899,2],0] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:321\
[grsacc07:01602] [[58899,2],0] grpcomm:base:modex: performing modex\
[grsacc07:01602] [[58899,2],0] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc07:01602] [[58899,2],0] grpcomm:base:full:modex: executing allgather\
[grsacc07:01604] [[58899,2],2] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:321\
[grsacc07:01604] [[58899,2],2] grpcomm:base:modex: performing modex\
[grsacc07:01604] [[58899,2],2] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc07:01604] [[58899,2],2] grpcomm:base:full:modex: executing allgather\
[grsacc07:01604] [[58899,2],2] grpcomm:bad entering allgather\
[grsacc07:01604] [[58899,2],2] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:311\
[grsacc07:01604] [[58899,2],2] grpcomm:base:modex: modex posted\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 3 to our daemon\
[grsacc07:01604] [[58899,2],2] grpcomm:bad allgather underway\
[grsacc07:01601] [[58899,0],1] COLLECTIVE RECVD FROM [[58899,2],0]\
[grsacc07:01601] [[58899,0],1] WORKING COLLECTIVE 3\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLLECTIVE 3\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 0\
[grsacc07:01601] [[58899,0],1] COLL 0 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 1\
[grsacc07:01601] [[58899,0],1] COLL 1 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 2\
[grsacc07:01601] [[58899,0],1] COLL 2 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 3\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 4\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 5\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19456] [[58899,2],6] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:321\
[grsacc09:19456] [[58899,2],6] grpcomm:base:modex: performing modex\
[grsacc09:19456] [[58899,2],6] grpcomm:base:pack_modex: reporting 2 entries\
[grsacc09:19456] [[58899,2],6] grpcomm:base:full:modex: executing allgather\
[grsacc09:19456] [[58899,2],6] grpcomm:bad entering allgather\
[grsacc09:19456] [[58899,2],6] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:311\
[grsacc09:19456] [[58899,2],6] grpcomm:base:modex: modex posted\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 3 to our daemon\
[grsacc09:19456] [[58899,2],6] grpcomm:bad allgather underway\
[grsacc09:19455] [[58899,0],3] COLLECTIVE RECVD FROM [[58899,2],7]\
[grsacc09:19455] [[58899,0],3] WORKING COLLECTIVE 3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLLECTIVE 3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 0\
[grsacc09:19455] [[58899,0],3] COLL 0 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 1\
[grsacc09:19455] [[58899,0],3] COLL 1 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 2\
[grsacc09:19455] [[58899,0],3] COLL 2 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 3\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 4\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 5\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] COLLECTIVE RECVD FROM [[58899,2],9]\
[grsacc09:19455] [[58899,0],3] COLLECTIVE RECVD FROM [[58899,2],8]\
[grsacc09:19455] [[58899,0],3] WORKING COLLECTIVE 3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLLECTIVE 3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 0\
[grsacc09:19455] [[58899,0],3] COLL 0 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 1\
[grsacc09:19455] [[58899,0],3] COLL 1 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 2\
[grsacc09:19455] [[58899,0],3] COLL 2 IS LOCALLY COMPLETE\
[grs[grsacc10:31295] [[58899,0],4] WORKING COLLECTIVE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLLECTIVE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 0\
[grsacc10:31295] [[58899,0],4] COLL 0 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGREacc09:19455] [[58899,0],3] PROGRESSING COLL id 3\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 4\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE SSING COLL id 1\
[grsacc10:31295] [[58899,0],4] COLL 1 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 2\
[grsacc10:31295] [[58899,0],4] COLL 2 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 3\
[grsacc10:31295] [[grsacc08:09178] [[58899,0],2] COLLECTIVE RECVD FROM [[58899,2],4]\
[grsacc08:09178] [[58899,0],2] WORKING COLLECTIVE 3\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLLECTIVE 3\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 0\
[grsacc08:09178] [[58899,0],3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 5\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 4\
[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 5\
[grsacc10:31[grsacc07:01601] [[58899,0],1] COLLECTIVE RECVD FROM [[58899,2],1]\
2] COLL 0 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 1\
[grsacc08:09178] [[58899,0],2] COLL 1 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 2\
[grsacc08:09178] [[58899,0],2] COLL 2 IS LOCALLY COMPLETE\
[grs295] [[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] WORKING COLLECTIVE 3\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLLECTIVE 3\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 0\
[grsacc07:01601] [[58899,0],1] COLL 0 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGREacc08:09178] [[58899,0],2] PROGRESSING COLL id 3\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 4\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE SSING COLL id 1\
[grsacc07:01601] [[58899,0],1] COLL 1 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 2\
[grsacc07:01601] [[58899,0],1] COLL 2 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 3\
[grsacc07:01601] [3\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 5\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] COLLECTIVE RECVD FROM [[58899,2],5]\
[grsacc08:09178] [[58899,0],2] WORKING COLLECTIVE 3\
[grs[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 4\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 5\
[grsacc07:01acc08:09178] [[58899,0],2] PROGRESSING COLLECTIVE 3\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 0\
[grsacc08:09178] [[58899,0],2] COLL 0 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 1\
[grsacc08:09178] [[58899,0],2] COLL 1 IS601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 2\
[grsacc08:09178] [[58899,0],2] COLL 2 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 3\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 4\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 5\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] COLLECTIVE RECVD FROM [[58899,2],11]\
[grsacc10:31295] [[58899,0],4] WORKING COLLECTIVE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLLECTIVE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 0\
[grsacc10:31295] [[58899,0],4] COLL 0 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 1\
[grsacc10:31295] [[58899,0],4] COLL 1 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 2\
[grsacc10:31295] [[58899,0],4] COLL 2 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 3\
[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 4\
[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 5\
[grsacc10:31[grsacc09:19455] [[58899,0],3] COLLECTIVE RECVD FROM [[58899,2],6]\
295] [[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] COLLECTIVE RECVD FROM [[58899,2],10]\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],3]\
[grsacc09:19455] [[58899,0],3] WORKING COLLECTIVE 3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLLECTIVE 3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 0\
[grsacc09:19455] [[58899,0],3] COLL 0 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRE[grsacc10:31295] [[58899,0],4] WORKING COLLECTIVE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLLECTIVE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 0\
[grsacc10:31295] [[58899,0],4] COLL 0 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRE[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: WORKING COLLECTIVE 3\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: CANNOT PROGRESS\
SSING COLL id 1\
[grsacc09:19455] [[58899,0],3] COLL 1 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 2\
[grsacc09:19455] [[58899,0],3] COLL 2 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 3\
[grsacc09:19455] [SSING COLL id 1\
[grsacc10:31295] [[58899,0],4] COLL 1 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 2\
[grsacc10:31295] [[58899,0],4] COLL 2 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 3\
[grsacc10:31295] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3] COLLECTIVE 3 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 4\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR [58899,0],4] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] COLLECTIVE 3 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 4\
[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FOR [grsacc07:01601] [[58899,0],1] COLLECTIVE RECVD FROM [[58899,2],2]\
JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 5\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],3JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 5\
[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],4[grsacc07:01601] [[58899,0],1] WORKING COLLECTIVE 3\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLLECTIVE 3\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 0\
[grsacc07:01601] [[58899,0],1] COLL 0 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRE]\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: WORKING COLLECTIVE 3\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],1]\
[grsacc09]\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: WORKING COLLECTIVE 3\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],0]\
[grsacc10SSING COLL id 1\
[grsacc07:01601] [[58899,0],1] COLL 1 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 2\
[grsacc07:01601] [[58899,0],1] COLL 2 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 3\
[grsacc07:01601] [[grsacc08:09178] [[58899,0],2] COLLECTIVE RECVD FROM [[58899,2],3]\
:19455] [[58899,0],3] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 3\
:31295] [[58899,0],4] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 3\
[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] COLLECTIVE 3 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 4\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR [grsacc08:09178] [[58899,0],2] WORKING COLLECTIVE 3\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLLECTIVE 3\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 0\
[grsacc08:09178] [[58899,0],2] COLL 0 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGREJOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 5\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],1SSING COLL id 1\
[grsacc08:09178] [[58899,0],2] COLL 1 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 2\
[grsacc08:09178] [[58899,0],2] COLL 2 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 3\
[grsacc08:09178] []\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: WORKING COLLECTIVE 3\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],0]\
[grsacc07[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] COLLECTIVE 3 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 4\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR :01601] [[58899,0],1] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 6\
JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 5\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],2]\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: WORKING COLLECTIVE 3\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],0]\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 3\
[grsacc07:01602] [[58899,2],0] grpcomm:bad entering allgather\
[grsacc07:01602] [[58899,2],0] ACTIVATING GRCPCOMM OP 3 at grpcomm_bad_module.c:311\
[grsacc07:01602] [[58899,2],0] grpcomm:base:modex: modex posted\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 3 to our daemon\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 3\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 3\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],4]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 3\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 6\
[grsacc07:01602] [[58899,2],0] grpcomm:bad allgather underway\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],1]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 3\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 6\
[grsacc06:19901] [[58899,0],0] grpcomm:bad:xcast sent to job [58899,2] tag 30\
[grsacc06:19901] [[58899,0],0] grpcomm:xcast:recv: with 1720 bytes\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],1]\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],4]\
[grsacc08:09178] [[58899,0],2] grpcomm:xcast:recv: with 1720 bytes\
[grsacc07:01601] [[58899,0],1] grpcomm:xcast:recv: with 1720 bytes\
[grsacc08:09178] [[58899,0],2] orte:daemon:send_relay - recipient list is empty!\
[grsacc07:01601] [[58899,0],1] orte:daemon:send_relay sending relay msg to [[58899,0],3]\
[grsacc10:31295] [[58899,0],4] grpcomm:xcast:recv: with 1720 bytes\
[grsacc10:31295] [[58899,0],4] orte:daemon:send_relay - recipient list is empty!\
[grsacc09:19455] [[58899,0],3] grpcomm:xcast:recv: with 1720 bytes\
[grsacc09:19455] [[58899,0],3] orte:daemon:send_relay - recipient list is empty!\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 3 recvd from [[58899,0],4]\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 3\
[grsacc10:31296] [[58899,2],9] STORING MODEX DATA\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],4]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],5]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],3]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],9]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],11]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 3 recvd from [[58899,0],3]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],10]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 3\
[grsacc09:19458] [[58899,2],8] STORING MODEX DATA\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],4]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],5]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],3]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],9]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],11]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],10]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],7]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],8]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],7]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],8]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],6]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 3 recvd from [[58899,0],3]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 3\
[grsacc09:19457] [[58899,2],7] STORING MODEX DATA\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],4]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],5]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],3]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],9]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],11]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],0]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],10]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],7]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],8]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],6]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],0]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 3 recvd from [[58899,0],3]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 3\
[grsacc09:19456] [[58899,2],6] STORING MODEX DATA\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],4]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],5]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],3]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],9]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],11]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],10]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],7]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],8]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],6]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],0]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],1]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],6]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],0]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],1]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],2]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc09:19458] [[58899,2],8] store:peer:modex[grsacc07:01601] [[58899,0],1] COLLECTIVE RECVD FROM [[58899,2],0]\
[grsacc07:01601] [[58899,0],1] WORKING COLLECTIVE 4\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLLECTIVE 4\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 0\
[grsacc07:01601] [[58899,0],1] COLL 0 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 1\
[grsacc07:01601] [[58899,0],1] COLL 1 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 2\
[grsacc07:01601] [[58899,0],1] COLL 2 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 3\
[grsacc07:01601] [[58899,0],1] COLL 3 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 4\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 5\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
NO MODEX RELEASE CBFUNC\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],2]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc09:19456] [[58899,2],6] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],1]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],2]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc09:19457] [[58899,2],7] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],1]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],2]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc10:31296] [[58899,2],9] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 3 recvd from [[58899,0],4]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 3\
[grsacc10:31298] [[58899,2],11] STORING MODEX DATA\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],4]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],5]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 3 recvd from [[58899,0],1]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 3\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc07:01603] [[58899,2],1] STORING MODEX DATA\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],4]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],3]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],9]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],5]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],11]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],3]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],10]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],9]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],7]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],8]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],11]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],10]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 3 recvd from [[58899,0],2]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],6]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],7]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 3\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 3 recvd from [[58899,0],2]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 3\
[grsacc08:09180] [[58899,2],4] STORING MODEX DATA\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],4]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],0]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],8]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 3 recvd from [[58899,0],2]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 3\
[grsacc08:09181] [[58899,2],5] STORING MODEX DATA\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],4]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],5]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],1]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],6]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],5]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],3]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],3]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],9]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],11]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],2]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],0]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],9]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],11]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],10]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],7]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],8]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc10:31298] [[58899,2],11] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],10]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],7]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],8]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 3 recvd from [[58899,0],4]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],6]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],0]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],1]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],1]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],6]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],0]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 3\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],2]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc08:09181] [[58899,2],5] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc10:31297] [[58899,2],10] STORING MODEX DATA\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],2]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],1]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],2]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc08:09180] [[58899,2],4] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],4]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc08:09179] [[58899,2],3] STORING MODEX DATA\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc07:01603] [[58899,2],1] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],4]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],5]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 3 recvd from [[58899,0],1]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 3\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],5]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],3]\
[grsacc07:01602] [[58899,2],0] STORING MODEX DATA\
[grsacc09:19457] [[58899,2],7] grpcomm:bad entering barrier\
[grsacc09:19457] [[58899,2],7] ACTIVATING GRCPCOMM OP 4 at grpcomm_bad_module.c:206\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],4]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad entering barrier\
[grsacc09:19456] [[58899,2],6] ACTIVATING GRCPCOMM OP 4 at grpcomm_bad_module.c:206\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],3]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],9]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc09:19458] [[58899,2],8] grpcomm:bad entering barrier\
[grsacc09:19458] [[58899,2],8] ACTIVATING GRCPCOMM OP 4 at grpcomm_bad_module.c:206\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],5]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad barrier underway\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],9]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],11]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad barrier underway\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],3]\
[grsacc09:19458] [[58899,2],8] grpcomm:bad barrier underway\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],11]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],10]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],9]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding [grsacc10:31295] [[58899,0],4] COLLECTIVE RECVD FROM [[58899,2],10]\
[grsacc10:31295] [[58899,0],4] WORKING COLLECTIVE 4\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLLECTIVE 4\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 0\
[grsacc10:31295] [[58899,0],4] COLL 0 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 1\
[grsacc10:31295] [[58899,0],4] COLL 1 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 2\
[grsacc10:31295] [[58899,0],4] COLL 2 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 3\
[grsacc10:31295] [[58899,0],4] COLL 3 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 4\
[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 5\
[grsacc10:31295] [[58899,0],4] A[grsacc09:19455] [[58899,0],3] COLLECTIVE RECVD FROM [[58899,2],8]\
[grsacc09:19455] [[58899,0],3] WORKING COLLECTIVE 4\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLLECTIVE 4\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 0\
[grsacc09:19455] [[58899,0],LL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] COLLECTIVE RECVD FROM [[58899,2],9]\
[grsacc10:31295] [[58899,0],4] WORKING COLLECTIVE 4\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLLECTIVE 4\
[grsacc10:31295] [[58899,0],4] PROG3] COLL 0 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 1\
[grsacc09:19455] [[58899,0],3] COLL 1 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 2\
[grsacc09:19455] [[58899,0],3] COLL 2 IS LOCALLY COMPLETE\
[grsRESSING COLL id 0\
[grsacc10:31295] [[58899,0],4] COLL 0 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 1\
[grsacc10:31295] [[58899,0],4] COLL 1 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 2\
[grsacc10:31295][grsacc08:09178] [[58899,0],2] COLLECTIVE RECVD FROM [[58899,2],3]\
acc09:19455] [[58899,0],3] PROGRESSING COLL id 3\
[grsacc09:19455] [[58899,0],3] COLL 3 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 4\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [ [[58899,0],4] COLL 2 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 3\
[grsacc10:31295] [[58899,0],4] COLL 3 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 4\
[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FOR[grsacc07:01601] [[58899,0],1] COLLECTIVE RECVD FROM [[58899,2],2]\
[grsacc07:01601] [[58899,0],1] WORKING COLLECTIVE 4\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLLECTIVE 4\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 0\
[grsacc07:01601] [[58899,0],[grsacc08:09178] [[58899,0],2] WORKING COLLECTIVE 4\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLLECTIVE 4\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 0\
[grsacc08:09178] [[58899,0],2] COLL 0 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRE[58899,0],3] PROGRESSING COLL id 5\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3] COLLECTIVE RECVD FROM [[58899,2],6]\
[grsacc09:19455] [[58899,0],3] WORKING COLLECTIVE 4\
[grsacc09:19455] [[58899 JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 5\
[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] COLLECTIVE RECVD FROM [[58899,2],11]\
[grsacc10:31295] [[58899,0],41] COLL 0 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 1\
[grsacc07:01601] [[58899,0],1] COLL 1 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 2\
[grsacc07:01601] [[58899,0],1] COLL 2 IS LOCALLY COMPLETE\
[grsSSING COLL id 1\
[grsacc08:09178] [[58899,0],2] COLL 1 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 2\
[grsacc08:09178] [[58899,0],2] COLL 2 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 3\
[grsacc08:09178] [,0],3] PROGRESSING COLLECTIVE 4\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 0\
[grsacc09:19455] [[58899,0],3] COLL 0 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 1\
[grsacc09:19455] [[58899,0],3] COLL 1 IS LOCALLY COMPLETE\
[g] WORKING COLLECTIVE 4\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLLECTIVE 4\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 0\
[grsacc10:31295] [[58899,0],4] COLL 0 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 1\
[grsacc10:312acc07:01601] [[58899,0],1] PROGRESSING COLL id 3\
[grsacc07:01601] [[58899,0],1] COLL 3 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 4\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],2] COLL 3 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 4\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 5\
[grsacc08:09178] [[58899,0],2] Arsacc09:19455] [[58899,0],3] PROGRESSING COLL id 2\
[grsacc09:19455] [[58899,0],3] COLL 2 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 3\
[grsacc09:19455] [[58899,0],3] COLL 3 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGR95] [[58899,0],4] COLL 1 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 2\
[grsacc10:31295] [[58899,0],4] COLL 2 IS LOCALLY COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 3\
[grsacc10:31295] [[58899,0],4] COLL 3 IS LOCALL[58899,0],1] PROGRESSING COLL id 5\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] COLLECTIVE RECVD FROM [[58899,2],1]\
[grsacc07:01601] [[58899,0],1] WORKING COLLECTIVE 4\
[grsacc07:01601] [[58899LL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] COLLECTIVE RECVD FROM [[58899,2],4]\
[grsacc08:09178] [[58899,0],2] WORKING COLLECTIVE 4\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLLECTIVE 4\
[grsacc08:09178] [[58899,0],2] PROGESSING COLL id 4\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 5\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3]Y COMPLETE\
[grsacc10:31295] [[58899,0],4] PROGRESSING COLL id 4\
[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] COLLECTIVE 4 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc10:31295] [[588,0],1] PROGRESSING COLLECTIVE 4\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 0\
[grsacc07:01601] [[58899,0],1] COLL 0 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 1\
[grsacc07:01601] [[58899,0],1] COLL 1 IS LOCALLY COMPLETE\
[gRESSING COLL id 0\
[grsacc08:09178] [[58899,0],2] COLL 0 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 1\
[grsacc08:09178] [[58899,0],2] COLL 1 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 2\
[grsacc08:09178] COLLECTIVE RECVD FROM [[58899,2],7]\
99,0],4] PROGRESSING COLL id 5\
[grsacc10:31295] [[58899,0],4] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],4]\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daersacc07:01601] [[58899,0],1] PROGRESSING COLL id 2\
[grsacc07:01601] [[58899,0],1] COLL 2 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 3\
[grsacc07:01601] [[58899,0],1] COLL 3 IS LOCALLY COMPLETE\
[grsacc07:01601] [[58899,0],1] PROGR [[58899,0],2] COLL 2 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 3\
[grsacc08:09178] [[58899,0],2] COLL 3 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 4\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR[grsacc09:19455] [[58899,0],3] WORKING COLLECTIVE 4\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLLECTIVE 4\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 0\
[grsacc09:19455] [[58899,0],3] COLL 0 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGREmon_coll: WORKING COLLECTIVE 4\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],0]\
[grsacc10:31295] [[58899,0],4] grpcomm:base:daemon_coll: MESSING COLL id 4\
[grsacc07:01601] [[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] COLLECTIVE 4 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc07:01601] [[58899,0],1] PROGRESSING COLL id 5\
[grsacc07:01601] JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 5\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] COLLECTIVE RECVD FROM [[58899,2],5]\
[grsacc08:09178] [[58899,0],2]SSING COLL id 1\
[grsacc09:19455] [[58899,0],3] COLL 1 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 2\
[grsacc09:19455] [[58899,0],3] COLL 2 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 3\
[grsacc09:19455] [ISSING CONTRIBUTORS: np 12 ngr 3\
[[58899,0],1] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],1]\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: WORKING COLLECTIVE 4\
[grsacc07:01601] WORKING COLLECTIVE 4\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLLECTIVE 4\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 0\
[grsacc08:09178] [[58899,0],2] COLL 0 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 1\
[grsacc08:0917[58899,0],3] COLL 3 IS LOCALLY COMPLETE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 4\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3] COLLECTIVE 4 LOCALLY COMPLETE - SENDING TO GLOBAL COLL [[58899,0],1] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 3\
8] [[58899,0],2] COLL 1 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 2\
[grsacc08:09178] [[58899,0],2] COLL 2 IS LOCALLY COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 3\
[grsacc08:09178] [[58899,0],2] COLL 3 IS LOCALLYECTIVE\
[grsacc09:19455] [[58899,0],3] PROGRESSING COLL id 5\
[grsacc09:19455] [[58899,0],3] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],3]\
[grsacc09:19455] [ COMPLETE\
[grsacc08:09178] [[58899,0],2] PROGRESSING COLL id 4\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] COLLECTIVE 4 LOCALLY COMPLETE - SENDING TO GLOBAL COLLECTIVE\
[grsacc08:09178] [[5889[58899,0],3] grpcomm:base:daemon_coll: WORKING COLLECTIVE 4\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc09:19455] [[58899,0],3] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],1]\
[grsacc09:19455] [[58899,0],39,0],2] PROGRESSING COLL id 5\
[grsacc08:09178] [[58899,0],2] ALL LOCAL PROCS FOR JOB [58899,2] CONTRIBUTE 3\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],2]\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daem] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 3\
on_coll: WORKING COLLECTIVE 4\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],0]\
[grsacc08:09178] [[58899,0],2] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 3\
2 entries for proc [[58899,2],10]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],7]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],11]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],7]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],8]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],10]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],8]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],6]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],7]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],6]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],0]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],8]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],0]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],1]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],6]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],1]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],0]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],2]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],1]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],2]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc07:01602] [[58899,2],0] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 3 recvd from [[58899,0],1]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 3\
[grsacc07:01604] [[58899,2],2] STORING MODEX DATA\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],4]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],5]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],3]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],9]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],11]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],10]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],7]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],8]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],6]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],0]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],2]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc10:31297] [[58899,2],10] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc10:31298] [[58899,2],11] grpcomm:bad entering barrier\
[grsacc10:31298] [[58899,2],11] ACTIVATING GRCPCOMM OP 4 at grpcomm_bad_module.c:206\
[grsacc10:31297] [[58899,2],10] grpcomm:bad entering barrier\
[grsacc10:31297] [[58899,2],10] ACTIVATING GRCPCOMM OP 4 at grpcomm_bad_module.c:206\
[grsacc10:31296] [[58899,2],9] grpcomm:bad entering barrier\
[grsacc10:31296] [[58899,2],9] ACTIVATING GRCPCOMM OP 4 at grpcomm_bad_module.c:206\
[grsacc10:31297] [[58899,2],10] grpcomm:bad barrier underway\
[grsacc10:31298] [[58899,2],11] grpcomm:bad barrier underway\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],1]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 2 entries for proc [[58899,2],2]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc07:01604] [[58899,2],2] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc07:01603] [[58899,2],1] grpcomm:bad entering barrier\
[grsacc07:01603] [[58899,2],1] ACTIVATING GRCPCOMM OP 4 at grpcomm_bad_module.c:206\
[grsacc07:01604] [[58899,2],2] grpcomm:bad entering barrier\
[grsacc07:01604] [[58899,2],2] ACTIVATING GRCPCOMM OP 4 at grpcomm_bad_module.c:206\
[grsacc07:01602] [[58899,2],0] grpcomm:bad entering barrier\
[grsacc07:01602] [[58899,2],0] ACTIVATING GRCPCOMM OP 4 at grpcomm_bad_module.c:206\
[grsacc07:01604] [[58899,2],2] grpcomm:bad barrier underway\
[grsacc10:31296] [[58899,2],9] grpcomm:bad barrier underway\
[grsacc07:01603] [[58899,2],1] grpcomm:bad barrier underway\
[grsacc08:09179] [[58899,2],3] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],4]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 4\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 3\
[grsacc08:09180] [[58899,2],4] grpcomm:bad entering barrier\
[grsacc08:09180] [[58899,2],4] ACTIVATING GRCPCOMM OP 4 at grpcomm_bad_module.c:206\
[grsacc07:01602] [[58899,2],0] grpcomm:bad barrier underway\
[grsacc08:09181] [[58899,2],5] grpcomm:bad entering barrier\
[grsacc08:09181] [[58899,2],5] ACTIVATING GRCPCOMM OP 4 at grpcomm_bad_module.c:206\
[grsacc08:09179] [[58899,2],3] grpcomm:bad entering barrier\
[grsacc08:09179] [[58899,2],3] ACTIVATING GRCPCOMM OP 4 at grpcomm_bad_module.c:206\
[grsacc08:09181] [[58899,2],5] grpcomm:bad barrier underway\
[grsacc08:09180] [[58899,2],4] grpcomm:bad barrier underway\
[grsacc08:09179] [[58899,2],3] grpcomm:bad barrier underway\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 4\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 6\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],3]\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: WORKING COLLECTIVE 4\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: NUM CONTRIBS: 3\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: RELAYING COLLECTIVE TO [[58899,0],0]\
[grsacc07:01601] [[58899,0],1] grpcomm:base:daemon_coll: MISSING CONTRIBUTORS: np 12 ngr 6\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: daemon collective recvd from [[58899,0],1]\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: WORKING COLLECTIVE 4\
[grsacc06:19901] [[58899,0],0] grpcomm:base:daemon_coll: NUM CONTRIBS: 6\
[grsacc06:19901] [[58899,0],0] grpcomm:bad:xcast sent to job [58899,2] tag 30\
[grsacc06:19901] [[58899,0],0] grpcomm:xcast:recv: with 40 bytes\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],1]\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],2]\
[grsacc06:19901] [[58899,0],0] orte:daemon:send_relay sending relay msg to [[58899,0],4]\
[grsacc08:09178] [[58899,0],2] grpcomm:xcast:recv: with 40 bytes\
[grsacc08:09178] [[58899,0],2] orte:daemon:send_relay - recipient list is empty!\
[grsacc10:31295] [[58899,0],4] grpcomm:xcast:recv: with 40 bytes\
[grsacc10:31295] [[58899,0],4] orte:daemon:send_relay - recipient list is empty!\
[grsacc07:01601] [[58899,0],1] grpcomm:xcast:recv: with 40 bytes\
[grsacc07:01601] [[58899,0],1] orte:daemon:send_relay sending relay msg to [[58899,0],3]\
[grsacc09:19455] [[58899,0],3] grpcomm:xcast:recv: with 40 bytes\
[grsacc09:19455] [[58899,0],3] orte:daemon:send_relay - recipient list is empty!\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 4 recvd from [[58899,0],2]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 4 recvd from [[58899,0],2]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 4\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 4 recvd from [[58899,0],2]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 4\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 4\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 4 recvd from [[58899,0],4]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 4\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 4 recvd from [[58899,0],4]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 4\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 4 recvd from [[58899,0],3]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 4 recvd from [[58899,0],3]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 4\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 4 recvd from [[58899,0],4]\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 4\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 4\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 4 recvd from [[58899,0],3]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 4\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 4 recvd from [[58899,0],1]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 4\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 4 recvd from [[58899,0],1]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 4\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 4 recvd from [[58899,0],1]\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 4\
[grsacc06:19901] [[58899,0],0] grpcomm:base:receive proc [[58899,2],0] requested coll id - returned id 6\
[grsacc06:19905] [[58899,1],1] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc06:19904] [[58899,1],0] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc06:19906] [[58899,1],2] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc06:19904] [[58899,1],0] grpcomm:base:modex: performing modex\
[grsacc06:19904] [[58899,1],0] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc06:19904] [[58899,1],0] grpcomm:base:full:modex: executing allgather\
[grsacc06:19904] [[58899,1],0] grpcomm:bad entering allgather\
[grsacc06:19904] [[58899,1],0] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc06:19904] [[58899,1],0] grpcomm:base:modex: modex posted\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc06:19904] [[58899,1],0] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc09:19458] [[58899,2],8] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc09:19456] [[58899,2],6] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc10:31297] [[58899,2],10] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc07:01604] [[58899,2],2] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc09:19458] [[58899,2],8] grpcomm:base:modex: performing modex\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc09:19458] [[58899,2],8] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc09:19458] [[58899,2],8] grpcomm:base:full:modex: executing allgather\
[grsacc09:19458] [[58899,2],8] grpcomm:bad entering allgather\
[grsacc09:19458] [[58899,2],8] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc09:19458] [[58899,2],8] grpcomm:base:modex: modex posted\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc10:31296] [[58899,2],9] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc07:01602] [[58899,2],0] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc08:09180] [[58899,2],4] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc08:09180] [[58899,2],4] grpcomm:base:modex: performing modex\
[grsacc10:31296] [[58899,2],9] grpcomm:base:modex: performing modex\
[grsacc07:01603] [[58899,2],1] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc09:19458] [[58899,2],8] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc08:09181] [[58899,2],5] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] grpcomm:base:modex: performing modex\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc08:09180] [[58899,2],4] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc08:09180] [[58899,2],4] grpcomm:base:full:modex: executing allgather\
[grsacc08:09180] [[58899,2],4] grpcomm:bad entering allgather\
[grsacc08:09180] [[58899,2],4] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc08:09180] [[58899,2],4] grpcomm:base:modex: modex posted\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc08:09180] [[58899,2],4] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:full:modex: executing allgather\
[grsacc07:01602] [[58899,2],0] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc08:09181] [[58899,2],5] grpcomm:base:modex: performing modex\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc08:09181] [[58899,2],5] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc08:09181] [[58899,2],5] grpcomm:base:full:modex: executing allgather\
[grsacc08:09181] [[58899,2],5] grpcomm:bad entering allgather\
[grsacc10:31298] [[58899,2],11] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc07:01602] [[58899,2],0] grpcomm:base:full:modex: executing allgather\
[grsacc09:19457] [[58899,2],7] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc08:09179] [[58899,2],3] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:321\
[grsacc10:31296] [[58899,2],9] grpcomm:bad entering allgather\
[grsacc07:01602] [[58899,2],0] grpcomm:bad entering allgather\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc08:09181] [[58899,2],5] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc08:09181] [[58899,2],5] grpcomm:base:modex: modex posted\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc10:31296] [[58899,2],9] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc07:01602] [[58899,2],0] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc09:19456] [[58899,2],6] grpcomm:base:modex: performing modex\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc09:19456] [[58899,2],6] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:base:modex: modex posted\
[grsacc07:01602] [[58899,2],0] grpcomm:base:modex: modex posted\
[grsacc09:19456] [[58899,2],6] grpcomm:base:full:modex: executing allgather\
[grsacc09:19456] [[58899,2],6] grpcomm:bad entering allgather\
[grsacc09:19456] [[58899,2],6] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc09:19456] [[58899,2],6] grpcomm:base:modex: modex posted\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:modex: performing modex\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc08:09179] [[58899,2],3] grpcomm:base:full:modex: executing allgather\
[grsacc08:09179] [[58899,2],3] grpcomm:bad entering allgather\
[grsacc08:09179] [[58899,2],3] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc08:09179] [[58899,2],3] grpcomm:base:modex: modex posted\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc09:19456] [[58899,2],6] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc08:09179] [[58899,2],3] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:modex: performing modex\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:full:modex: executing allgather\
[grsacc09:19457] [[58899,2],7] grpcomm:bad entering allgather\
[grsacc09:19457] [[58899,2],7] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc09:19457] [[58899,2],7] grpcomm:base:modex: modex posted\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc09:19457] [[58899,2],7] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc07:01602] [[58899,2],0] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc10:31296] [[58899,2],9] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:modex: performing modex\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc07:01604] [[58899,2],2] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc07:01604] [[58899,2],2] grpcomm:base:full:modex: executing allgather\
[grsacc07:01604] [[58899,2],2] grpcomm:bad entering allgather\
[grsacc07:01604] [[58899,2],2] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc07:01604] [[58899,2],2] grpcomm:base:modex: modex posted\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc08:09181] [[58899,2],5] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc07:01604] [[58899,2],2] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:modex: performing modex\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc10:31297] [[58899,2],10] grpcomm:base:full:modex: executing allgather\
[grsacc10:31297] [[58899,2],10] grpcomm:bad entering allgather\
[grsacc10:31297] [[58899,2],10] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc10:31297] [[58899,2],10] grpcomm:base:modex: modex posted\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc10:31297] [[58899,2],10] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:base:modex: performing modex\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc07:01603] [[58899,2],1] grpcomm:base:modex: performing modex\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc07:01603] [[58899,2],1] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc07:01603] [[58899,2],1] grpcomm:base:full:modex: executing allgather\
[grsacc07:01603] [[58899,2],1] grpcomm:bad entering allgather\
[grsacc07:01603] [[58899,2],1] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc07:01603] [[58899,2],1] grpcomm:base:modex: modex posted\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc07:01603] [[58899,2],1] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:full:modex: executing allgather\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:bad entering allgather\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:modex: modex posted\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc06:19906] [[58899,1],2] grpcomm:base:modex: performing modex\
[grsacc06:19906] [[58899,1],2] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc06:19906] [[58899,1],2] grpcomm:base:full:modex: executing allgather\
[grsacc06:19906] [[58899,1],2] grpcomm:bad entering allgather\
[grsacc06:19906] [[58899,1],2] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc06:19906] [[58899,1],2] grpcomm:base:modex: modex posted\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc06:19906] [[58899,1],2] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:modex: performing modex\
[grsacc06:19905] [[58899,1],1] grpcomm:base:pack_modex: reporting 5 entries\
[grsacc06:19905] [[58899,1],1] grpcomm:base:full:modex: executing allgather\
[grsacc06:19905] [[58899,1],1] grpcomm:bad entering allgather\
[grsacc06:19905] [[58899,1],1] ACTIVATING GRCPCOMM OP 6 at grpcomm_bad_module.c:311\
[grsacc06:19905] [[58899,1],1] grpcomm:base:modex: modex posted\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,1],0]\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,1],1]\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,1],2]\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,2],0]\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,2],1]\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,2],2]\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,2],3]\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,2],4]\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,2],5]\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,2],6]\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,2],7]\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,2],8]\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,2],9]\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,2],10]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:bad sending collective 6 to [[58899,2],11]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],0]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],8]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],4]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],3]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],6]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],9]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],0]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],5]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],10]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],2]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],7]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],1]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,2],11]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc06:19904] [[58899,1],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc06:19904] [[58899,1],0] CHECKING COLL id 6\
[grsacc06:19904] [[58899,1],0] STORING MODEX DATA\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc06:19904] [[58899,1],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc06:19904] [[58899,1],0] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc06:19904] [[58899,1],0] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc08:09179] [[58899,2],3] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc08:09179] [[58899,2],3] CHECKING COLL id 6\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc06:19905] [[58899,1],1] CHECKING COLL id 6\
[grsacc06:19905] [[58899,1],1] STORING MODEX DATA\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc08:09179] [[58899,2],3] STORING MODEX DATA\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc07:01603] [[58899,2],1] CHECKING COLL id 6\
[grsacc07:01603] [[58899,2],1] STORING MODEX DATA\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc06:19905] [[58899,1],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc06:19905] [[58899,1],1] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc06:19905] [[58899,1],1] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc08:09179] [[58899,2],3] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc08:09179] [[58899,2],3] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc08:09179] [[58899,2],3] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc07:01603] [[58899,2],1] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc07:01603] [[58899,2],1] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc07:01603] [[58899,2],1] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc08:09180] [[58899,2],4] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc08:09180] [[58899,2],4] CHECKING COLL id 6\
[grsacc08:09180] [[58899,2],4] STORING MODEX DATA\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc09:19456] [[58899,2],6] CHECKING COLL id 6\
[grsacc09:19456] [[58899,2],6] STORING MODEX DATA\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc10:31296] [[58899,2],9] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc10:31296] [[58899,2],9] STORING MODEX DATA\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc09:19456] [[58899,2],6] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc09:19456] [[58899,2],6] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc09:19456] [[58899,2],6] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc08:09180] [[58899,2],4] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc08:09180] [[58899,2],4] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc08:09180] [[58899,2],4] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc09:19457] [[58899,2],7] CHECKING COLL id 6\
[grsacc09:19457] [[58899,2],7] STORING MODEX DATA\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc08:09181] [[58899,2],5] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc08:09181] [[58899,2],5] CHECKING COLL id 6\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc07:01602] [[58899,2],0] CHECKING COLL id 6\
[grsacc07:01602] [[58899,2],0] STORING MODEX DATA\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc08:09181] [[58899,2],5] STORING MODEX DATA\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc09:19457] [[58899,2],7] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc09:19457] [[58899,2],7] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc09:19457] [[58899,2],7] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc09:19458] [[58899,2],8] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc09:19458] [[58899,2],8] CHECKING COLL id 6\
[grsacc09:19458] [[58899,2],8] STORING MODEX DATA\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc08:09181] [[58899,2],5] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc08:09181] [[58899,2],5] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc08:09181] [[58899,2],5] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc09:19458] [[58899,2],8] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc09:19458] [[58899,2],8] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc09:19458] [[58899,2],8] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc10:31296] [[58899,2],9] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc10:31296] [[58899,2],9] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc10:31296] [[58899,2],9] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc07:01602] [[58899,2],0] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc07:01602] [[58899,2],0] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc07:01602] [[58899,2],0] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc10:31297] [[58899,2],10] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] STORING MODEX DATA\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc07:01604] [[58899,2],2] CHECKING COLL id 6\
[grsacc07:01604] [[58899,2],2] STORING MODEX DATA\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc10:31297] [[58899,2],10] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc10:31297] [[58899,2],10] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc07:01604] [[58899,2],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc10:31297] [[58899,2],10] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],2]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc10:31298] [[58899,2],11] CHECKING COLL id 6\
[grsacc10:31298] [[58899,2],11] STORING MODEX DATA\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc10:31298] [[58899,2],11] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc10:31298] [[58899,2],11] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc10:31298] [[58899,2],11] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc07:01604] [[58899,2],2] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc07:01604] [[58899,2],2] store:peer:modex NO MODEX RELEASE CBFUNC\
[grsacc06:19906] [[58899,1],2] grpcomm:base:receive processing collective return for id 6 recvd from [[58899,1],1]\
[grsacc06:19906] [[58899,1],2] CHECKING COLL id 6\
[grsacc06:19906] [[58899,1],2] STORING MODEX DATA\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],0]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,1],0]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],2]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,1],2]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],8]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,2],8]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],4]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,2],4]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],3]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,2],3]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],9]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,2],9]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],6]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,2],6]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],0]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,2],0]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],5]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,2],5]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],2]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,2],2]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],7]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,2],7]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],10]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,2],10]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],1]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,2],1]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,2],11]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,2],11]\
[grsacc06:19906] [[58899,1],2] grpcomm:base:update_modex_entries: adding 5 entries for proc [[58899,1],1]\
[grsacc06:19906] [[58899,1],2] store:peer:modex: completed modex entry for proc [[58899,1],1]\
[grsacc06:19906] [[58899,1],2] store:peer:modex NO MODEX RELEASE CBFUNC\
^C\
grsprabh@grsacc06:~/resmgmt/branches/Malleability/workdir$ [grsacc07:01601] [[58899,0],1] grpcomm:base:receive stop comm\
[grsacc10:31295] [[58899,0],4] grpcomm:base:receive stop comm\
[grsacc10:31295] [[58899,0],4] plm:base:receive stop comm\
[grsacc07:01601] [[58899,0],1] plm:base:receive stop comm\
[grsacc08:09178] [[58899,0],2] grpcomm:base:receive stop comm\
[grsacc08:09178] [[58899,0],2] plm:base:receive stop comm\
[grsacc09:19455] [[58899,0],3] grpcomm:base:receive stop comm\
[grsacc09:19455] [[58899,0],3] plm:base:receive stop comm}
On Feb 21, 2014, at 5:03 PM, Ralph Castain wrote:
> Well, that all looks fine. However, I note that the procs on grsacc04 all
> stopped making progress at the same point, which is why the job hung. All the
> procs on the other nodes were just fine.
>
> So let's try a couple of things:
>
> 1. add "--bind-to none" to your cmd line so we avoid any contention issues
>
> 2. if possible, remove grsacc04 from the allocation (you can just use the
> -host option on the cmd line to ignore it), and/or replace that host with
> another one. Let's see if the problem has something to do with that specific
> node.
>
>
> On Feb 21, 2014, at 4:08 AM, Suraj Prabhakaran <suraj.prabhaka...@gmail.com>
> wrote:
>
>> Right, so I have the output here. Same case,
>>
>> mpiexec -mca plm_base_verbose 5 -mca ess_base_verbose 5 -mca
>> grpcomm_base_verbose 5 -np 3 ./simple_spawn
>>
>> Output attached.
>>
>> Best,
>> Suraj
>>
>> <output>
>>
>> On Feb 21, 2014, at 5:30 AM, Ralph Castain wrote:
>>
>>>
>>> On Feb 20, 2014, at 7:05 PM, Suraj Prabhakaran
>>> <suraj.prabhaka...@gmail.com> wrote:
>>>
>>>> Thanks Ralph!
>>>>
>>>> I must have mentioned though. Without the Torque environment, spawning
>>>> with ssh works ok. But Under the torque environment, not.
>>>
>>> Ah, no - you forgot to mention that point.
>>>
>>>>
>>>> I started the simple_spawn with 3 processes and spawned 9 processes (3 per
>>>> node on 3 nodes).
>>>>
>>>> There is no problem with the Torque environment because all the 9
>>>> processes are started on the respective nodes. But the MPI_Comm_spawn of
>>>> the parent and MPI_Init of the children, "sometimes" don't return!
>>>
>>> Seems odd - the launch environment has nothing to do with MPI_Init, so if
>>> the processes are indeed being started, they should run. One possibility is
>>> that they aren't correctly getting some wireup info.
>>>
>>> Can you configure OMPI --enable-debug and then rerun the example with "-mca
>>> plm_base_verbose 5 -mca ess_base_verbose 5 -mca grpcomm_base_verbose 5" on
>>> the command line?
>>>
>>>
>>>>
>>>> This is the output of simple_spawn - which confirms the above statement.
>>>>
>>>> [pid 31208] starting up!
>>>> [pid 31209] starting up!
>>>> [pid 31210] starting up!
>>>> 0 completed MPI_Init
>>>> Parent [pid 31208] about to spawn!
>>>> 1 completed MPI_Init
>>>> Parent [pid 31209] about to spawn!
>>>> 2 completed MPI_Init
>>>> Parent [pid 31210] about to spawn!
>>>> [pid 28630] starting up!
>>>> [pid 28631] starting up!
>>>> [pid 9846] starting up!
>>>> [pid 9847] starting up!
>>>> [pid 9848] starting up!
>>>> [pid 6363] starting up!
>>>> [pid 6361] starting up!
>>>> [pid 6362] starting up!
>>>> [pid 28632] starting up!
>>>>
>>>> Any hints?
>>>>
>>>> Best,
>>>> Suraj
>>>>
>>>> On Feb 21, 2014, at 3:44 AM, Ralph Castain wrote:
>>>>
>>>>> Hmmm...I don't see anything immediately glaring. What do you mean by
>>>>> "doesn't work"? Is there some specific behavior you see?
>>>>>
>>>>> You might try the attached program. It's a simple spawn test we use -
>>>>> 1.7.4 seems happy with it.
>>>>>
>>>>> <simple_spawn.c>
>>>>>
>>>>> On Feb 20, 2014, at 10:14 AM, Suraj Prabhakaran
>>>>> <suraj.prabhaka...@gmail.com> wrote:
>>>>>
>>>>>> I am using 1.7.4!
>>>>>>
>>>>>> On Feb 20, 2014, at 7:00 PM, Ralph Castain wrote:
>>>>>>
>>>>>>> What OMPI version are you using?
>>>>>>>
>>>>>>> On Feb 20, 2014, at 7:56 AM, Suraj Prabhakaran
>>>>>>> <suraj.prabhaka...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello!
>>>>>>>>
>>>>>>>> I am having problem using MPI_Comm_spawn under torque. It doesn't work
>>>>>>>> when spawning more than 12 processes on various nodes. To be more
>>>>>>>> precise, "sometimes" it works, and "sometimes" it doesn't!
>>>>>>>>
>>>>>>>> Here is my case. I obtain 5 nodes, 3 cores per node and my
>>>>>>>> $PBS_NODEFILE looks like below.
>>>>>>>>
>>>>>>>> node1
>>>>>>>> node1
>>>>>>>> node1
>>>>>>>> node2
>>>>>>>> node2
>>>>>>>> node2
>>>>>>>> node3
>>>>>>>> node3
>>>>>>>> node3
>>>>>>>> node4
>>>>>>>> node4
>>>>>>>> node4
>>>>>>>> node5
>>>>>>>> node5
>>>>>>>> node5
>>>>>>>>
>>>>>>>> I started a hello program (which just spawns itself and of course, the
>>>>>>>> children don't spawn), with
>>>>>>>>
>>>>>>>> mpiexec -np 3 ./hello
>>>>>>>>
>>>>>>>> Spawning 3 more processes (on node 2) - works!
>>>>>>>> spawning 6 more processes (node 2 and 3) - works!
>>>>>>>> spawning 9 processes (node 2,3,4) - "sometimes" OK, "sometimes" not!
>>>>>>>> spawning 12 processes (node 2,3,4,5) - "mostly" not!
>>>>>>>>
>>>>>>>> I ideally want to spawn about 32 processes with large number of nodes,
>>>>>>>> but this is at the moment impossible. I have attached my hello program
>>>>>>>> to this email.
>>>>>>>>
>>>>>>>> I will be happy to provide any more info or verbose outputs if you
>>>>>>>> could please tell me what exactly you would like to see.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Suraj
>>>>>>>>
>>>>>>>> <hello.c>_______________________________________________
>>>>>>>> devel mailing list
>>>>>>>> de...@open-mpi.org
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> devel mailing list
>>>>>>> de...@open-mpi.org
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing list
>>>>>> de...@open-mpi.org
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>
>>>>> _______________________________________________
>>>>> devel mailing list
>>>>> de...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> de...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>> _______________________________________________
>>> devel mailing list
>>> de...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> _______________________________________________
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
