> On 23 Oct 2015, at 23:45 , Mark Santcroos <mark.santcr...@rutgers.edu> wrote: > the second is output from my parser script
I figured you might want the output of the succeeded jobs too, please see the updated output attached.
Jobs started 16 Jobs completed: 15 Procs completed: 16 Communication "Errors": 15 JOB [9019,9] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) JOB [9019,11] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) JOB [9019,10] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) JOB [9019,16] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) JOB [9019,8] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) JOB [9019,12] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) JOB [9019,4] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) JOB [9019,6] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) JOB [9019,7] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) JOB [9019,2] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) JOB [9019,3] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING (7) JOB [9019,3] not complete!! (final state: RUNNING) JOB [9019,5] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) JOB [9019,13] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) JOB [9019,1] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) JOB [9019,15] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) JOB [9019,14] INIT_COMPLETE, ALLOCATION, ALL DAEMONS REPORTED, VM READY, MAP COMPLETE, LOCAL LAUNCH, RUNNING, NORMALLY TERMINATED, NOTIFY COMPLETED (9) PROC [[9019,7],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) PROC [[9019,13],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) PROC [[9019,11],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) PROC [[9019,5],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) PROC [[9019,16],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) PROC [[9019,10],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) PROC [[9019,3],0] RUNNING, IOF COMPLETE, KILLED BY INTERNAL COMMAND, WAITPID FIRED, NORMALLY TERMINATED (5) PROC [[9019,3],0] KILLED!! (final state: NORMALLY TERMINATED) PROC [[9019,14],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) PROC [[9019,9],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) PROC [[9019,12],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) PROC [[9019,15],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) PROC [[9019,1],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) PROC [[9019,8],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) PROC [[9019,2],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) PROC [[9019,4],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) PROC [[9019,6],0] RUNNING, IOF COMPLETE, WAITPID FIRED, NORMALLY TERMINATED (4) Log for succeeded job: [9019,9] ['ACTIVATE JOB [9019,9] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,9] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,9] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,9] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,9] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,9] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,9] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,9] STATE ALLOCATION COMPLETE PRI 4', 'ACTIVATE JOB [9019,9] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,9] STATE PENDING DAEMON LAUNCH PRI 4', 'state:base:check_job_completed job [9019,9] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,9] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'ACTIVATE JOB [9019,9] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,9] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,9] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,9] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,9] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,9] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,9] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,9] STATE MAP COMPLETE PRI 4', 'ACTIVATE JOB [9019,9] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,9] STATE PENDING FINAL SYSTEM PREP PRI 4', 'state:base:check_job_completed job [9019,9] is not terminated (0:1)', 'state:base:check_job_completed job [9019,9] is not terminated (0:1)', 'state:base:check_job_completed job [9019,9] is not terminated (0:1)', 'state:base:check_job_completed job [9019,9] is not terminated (0:1)', 'ACTIVATE JOB [9019,9] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,9] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,9] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,9] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,9] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,9] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,9] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'state:base:check_job_completed job [9019,9] is not terminated (0:1)', 'ACTIVATE JOB [9019,9] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,9] STATE NORMALLY TERMINATED PRI 4', 'state:base:check_job_completed job [9019,9] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,9] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,9] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,9] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,9] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,9] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,9] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'JOB [9019,9] HAS TERMINATED', 'ACTIVATE JOB [9019,9] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,9] STATE NOTIFY COMPLETED PRI 4'] Log for succeeded job: [9019,11] ['ACTIVATE JOB [9019,11] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,11] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,11] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,11] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,11] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,11] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,11] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,11] STATE ALLOCATION COMPLETE PRI 4', 'ACTIVATE JOB [9019,11] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,11] STATE PENDING DAEMON LAUNCH PRI 4', 'state:base:check_job_completed job [9019,11] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,11] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'ACTIVATE JOB [9019,11] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,11] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,11] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,11] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,11] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,11] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,11] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,11] STATE MAP COMPLETE PRI 4', 'ACTIVATE JOB [9019,11] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,11] STATE PENDING FINAL SYSTEM PREP PRI 4', 'state:base:check_job_completed job [9019,11] is not terminated (0:1)', 'state:base:check_job_completed job [9019,11] is not terminated (0:1)', 'state:base:check_job_completed job [9019,11] is not terminated (0:1)', 'state:base:check_job_completed job [9019,11] is not terminated (0:1)', 'ACTIVATE JOB [9019,11] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,11] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,11] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,11] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,11] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,11] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,11] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'state:base:check_job_completed job [9019,11] is not terminated (0:1)', 'ACTIVATE JOB [9019,11] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,11] STATE NORMALLY TERMINATED PRI 4', 'state:base:check_job_completed job [9019,11] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,11] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,11] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,11] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,11] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'JOB [9019,11] HAS TERMINATED', 'ACTIVATE JOB [9019,11] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,11] STATE NOTIFY COMPLETED PRI 4', 'state:base:check_job_completed job [9019,11] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,11] is terminated (1 vs 1 [NORMALLY TERMINATED])'] Log for succeeded job: [9019,10] ['ACTIVATE JOB [9019,10] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,10] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,10] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,10] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,10] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,10] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,10] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,10] STATE ALLOCATION COMPLETE PRI 4', 'ACTIVATE JOB [9019,10] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,10] STATE PENDING DAEMON LAUNCH PRI 4', 'state:base:check_job_completed job [9019,10] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,10] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'ACTIVATE JOB [9019,10] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,10] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,10] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,10] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,10] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,10] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,10] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,10] STATE MAP COMPLETE PRI 4', 'ACTIVATE JOB [9019,10] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,10] STATE PENDING FINAL SYSTEM PREP PRI 4', 'state:base:check_job_completed job [9019,10] is not terminated (0:1)', 'state:base:check_job_completed job [9019,10] is not terminated (0:1)', 'state:base:check_job_completed job [9019,10] is not terminated (0:1)', 'state:base:check_job_completed job [9019,10] is not terminated (0:1)', 'ACTIVATE JOB [9019,10] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,10] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,10] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,10] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,10] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,10] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,10] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'state:base:check_job_completed job [9019,10] is not terminated (0:1)', 'ACTIVATE JOB [9019,10] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,10] STATE NORMALLY TERMINATED PRI 4', 'state:base:check_job_completed job [9019,10] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,10] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,10] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,10] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,10] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,10] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'JOB [9019,10] HAS TERMINATED', 'ACTIVATE JOB [9019,10] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,10] STATE NOTIFY COMPLETED PRI 4', 'state:base:check_job_completed job [9019,10] is terminated (1 vs 1 [NORMALLY TERMINATED])'] Log for succeeded job: [9019,16] ['ACTIVATE JOB [9019,16] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,16] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,16] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,16] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,16] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,16] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,16] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,16] STATE ALLOCATION COMPLETE PRI 4', 'state:base:check_job_completed job [9019,16] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,16] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'ACTIVATE JOB [9019,16] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,16] STATE PENDING DAEMON LAUNCH PRI 4', 'ACTIVATE JOB [9019,16] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,16] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,16] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,16] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,16] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,16] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,16] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,16] STATE MAP COMPLETE PRI 4', 'state:base:check_job_completed job [9019,16] is not terminated (0:1)', 'state:base:check_job_completed job [9019,16] is not terminated (0:1)', 'state:base:check_job_completed job [9019,16] is not terminated (0:1)', 'state:base:check_job_completed job [9019,16] is not terminated (0:1)', 'ACTIVATE JOB [9019,16] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,16] STATE PENDING FINAL SYSTEM PREP PRI 4', 'ACTIVATE JOB [9019,16] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,16] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,16] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,16] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,16] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,16] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,16] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'state:base:check_job_completed job [9019,16] is not terminated (0:1)', 'ACTIVATE JOB [9019,16] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,16] STATE NORMALLY TERMINATED PRI 4', 'JOB [9019,16] HAS TERMINATED', 'ACTIVATE JOB [9019,16] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,16] STATE NOTIFY COMPLETED PRI 4', 'state:base:check_job_completed job [9019,16] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,16] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,16] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,16] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,16] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,16] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,16] is terminated (1 vs 1 [NORMALLY TERMINATED])'] Log for succeeded job: [9019,8] ['ACTIVATE JOB [9019,8] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,8] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,8] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,8] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,8] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,8] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,8] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,8] STATE ALLOCATION COMPLETE PRI 4', 'ACTIVATE JOB [9019,8] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,8] STATE PENDING DAEMON LAUNCH PRI 4', 'ACTIVATE JOB [9019,8] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,8] STATE ALL DAEMONS REPORTED PRI 4', 'state:base:check_job_completed job [9019,8] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,8] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'ACTIVATE JOB [9019,8] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,8] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,8] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,8] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,8] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,8] STATE MAP COMPLETE PRI 4', 'ACTIVATE JOB [9019,8] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,8] STATE PENDING FINAL SYSTEM PREP PRI 4', 'ACTIVATE JOB [9019,8] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,8] STATE PENDING APP LAUNCH PRI 4', 'state:base:check_job_completed job [9019,8] is not terminated (0:1)', 'state:base:check_job_completed job [9019,8] is not terminated (0:1)', 'state:base:check_job_completed job [9019,8] is not terminated (0:1)', 'state:base:check_job_completed job [9019,8] is not terminated (0:1)', 'ACTIVATE JOB [9019,8] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,8] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,8] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,8] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,8] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'ACTIVATE JOB [9019,8] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,8] STATE NORMALLY TERMINATED PRI 4', 'JOB [9019,8] HAS TERMINATED', 'ACTIVATE JOB [9019,8] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,8] STATE NOTIFY COMPLETED PRI 4'] Log for succeeded job: [9019,12] ['ACTIVATE JOB [9019,12] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,12] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,12] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,12] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,12] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,12] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,12] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,12] STATE ALLOCATION COMPLETE PRI 4', 'state:base:check_job_completed job [9019,12] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,12] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'ACTIVATE JOB [9019,12] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,12] STATE PENDING DAEMON LAUNCH PRI 4', 'ACTIVATE JOB [9019,12] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,12] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,12] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,12] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,12] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,12] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,12] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,12] STATE MAP COMPLETE PRI 4', 'state:base:check_job_completed job [9019,12] is not terminated (0:1)', 'state:base:check_job_completed job [9019,12] is not terminated (0:1)', 'state:base:check_job_completed job [9019,12] is not terminated (0:1)', 'state:base:check_job_completed job [9019,12] is not terminated (0:1)', 'ACTIVATE JOB [9019,12] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,12] STATE PENDING FINAL SYSTEM PREP PRI 4', 'ACTIVATE JOB [9019,12] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,12] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,12] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,12] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,12] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,12] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,12] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'state:base:check_job_completed job [9019,12] is not terminated (0:1)', 'ACTIVATE JOB [9019,12] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,12] STATE NORMALLY TERMINATED PRI 4', 'state:base:check_job_completed job [9019,12] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,12] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,12] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,12] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'JOB [9019,12] HAS TERMINATED', 'ACTIVATE JOB [9019,12] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,12] STATE NOTIFY COMPLETED PRI 4', 'state:base:check_job_completed job [9019,12] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,12] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,12] is terminated (1 vs 1 [NORMALLY TERMINATED])'] Log for succeeded job: [9019,4] ['ACTIVATE JOB [9019,4] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,4] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,4] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,4] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,4] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,4] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,4] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,4] STATE ALLOCATION COMPLETE PRI 4', 'ACTIVATE JOB [9019,4] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,4] STATE PENDING DAEMON LAUNCH PRI 4', 'ACTIVATE JOB [9019,4] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,4] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,4] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,4] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,4] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,4] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,4] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,4] STATE MAP COMPLETE PRI 4', 'ACTIVATE JOB [9019,4] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,4] STATE PENDING FINAL SYSTEM PREP PRI 4', 'ACTIVATE JOB [9019,4] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,4] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,4] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,4] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,4] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,4] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,4] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'state:base:check_job_completed job [9019,4] is not terminated (0:1)', 'state:base:check_job_completed job [9019,4] is not terminated (0:1)', 'ACTIVATE JOB [9019,4] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,4] STATE NORMALLY TERMINATED PRI 4', 'state:base:check_job_completed job [9019,4] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,4] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,4] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'JOB [9019,4] HAS TERMINATED', 'ACTIVATE JOB [9019,4] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,4] STATE NOTIFY COMPLETED PRI 4'] Log for succeeded job: [9019,6] ['ACTIVATE JOB [9019,6] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,6] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,6] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,6] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,6] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,6] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,6] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,6] STATE ALLOCATION COMPLETE PRI 4', 'ACTIVATE JOB [9019,6] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,6] STATE PENDING DAEMON LAUNCH PRI 4', 'ACTIVATE JOB [9019,6] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,6] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,6] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,6] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,6] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,6] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,6] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,6] STATE MAP COMPLETE PRI 4', 'ACTIVATE JOB [9019,6] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,6] STATE PENDING FINAL SYSTEM PREP PRI 4', 'ACTIVATE JOB [9019,6] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,6] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,6] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,6] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,6] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,6] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,6] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'state:base:check_job_completed job [9019,6] is not terminated (0:1)', 'state:base:check_job_completed job [9019,6] is not terminated (0:1)', 'ACTIVATE JOB [9019,6] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,6] STATE NORMALLY TERMINATED PRI 4', 'state:base:check_job_completed job [9019,6] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'JOB [9019,6] HAS TERMINATED', 'ACTIVATE JOB [9019,6] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,6] STATE NOTIFY COMPLETED PRI 4', 'state:base:check_job_completed job [9019,6] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,6] is terminated (1 vs 1 [NORMALLY TERMINATED])'] Log for succeeded job: [9019,7] ['ACTIVATE JOB [9019,7] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,7] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,7] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,7] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,7] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,7] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,7] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,7] STATE ALLOCATION COMPLETE PRI 4', 'ACTIVATE JOB [9019,7] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,7] STATE PENDING DAEMON LAUNCH PRI 4', 'ACTIVATE JOB [9019,7] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,7] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,7] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,7] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,7] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,7] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,7] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,7] STATE MAP COMPLETE PRI 4', 'ACTIVATE JOB [9019,7] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,7] STATE PENDING FINAL SYSTEM PREP PRI 4', 'ACTIVATE JOB [9019,7] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,7] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,7] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,7] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,7] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,7] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,7] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'state:base:check_job_completed job [9019,7] is not terminated (0:1)', 'state:base:check_job_completed job [9019,7] is not terminated (0:1)', 'ACTIVATE JOB [9019,7] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,7] STATE NORMALLY TERMINATED PRI 4', 'JOB [9019,7] HAS TERMINATED', 'ACTIVATE JOB [9019,7] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,7] STATE NOTIFY COMPLETED PRI 4', 'state:base:check_job_completed job [9019,7] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,7] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,7] is terminated (1 vs 1 [NORMALLY TERMINATED])'] Log for succeeded job: [9019,2] ['ACTIVATE JOB [9019,2] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,2] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,2] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,2] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,2] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,2] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,2] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,2] STATE ALLOCATION COMPLETE PRI 4', 'ACTIVATE JOB [9019,2] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,2] STATE PENDING DAEMON LAUNCH PRI 4', 'ACTIVATE JOB [9019,2] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,2] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,2] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,2] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,2] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,2] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,2] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,2] STATE MAP COMPLETE PRI 4', 'ACTIVATE JOB [9019,2] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,2] STATE PENDING FINAL SYSTEM PREP PRI 4', 'ACTIVATE JOB [9019,2] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,2] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,2] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,2] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,2] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,2] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,2] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'ACTIVATE JOB [9019,2] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,2] STATE NORMALLY TERMINATED PRI 4', 'JOB [9019,2] HAS TERMINATED', 'ACTIVATE JOB [9019,2] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,2] STATE NOTIFY COMPLETED PRI 4', 'state:base:check_job_completed job [9019,2] is terminated (1 vs 1 [NORMALLY TERMINATED])'] Log for succeeded job: [9019,5] ['ACTIVATE JOB [9019,5] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,5] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,5] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,5] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,5] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,5] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,5] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,5] STATE ALLOCATION COMPLETE PRI 4', 'ACTIVATE JOB [9019,5] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,5] STATE PENDING DAEMON LAUNCH PRI 4', 'ACTIVATE JOB [9019,5] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,5] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,5] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,5] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,5] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,5] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,5] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,5] STATE MAP COMPLETE PRI 4', 'ACTIVATE JOB [9019,5] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,5] STATE PENDING FINAL SYSTEM PREP PRI 4', 'ACTIVATE JOB [9019,5] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,5] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,5] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,5] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,5] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,5] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,5] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'state:base:check_job_completed job [9019,5] is not terminated (0:1)', 'state:base:check_job_completed job [9019,5] is not terminated (0:1)', 'ACTIVATE JOB [9019,5] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,5] STATE NORMALLY TERMINATED PRI 4', 'state:base:check_job_completed job [9019,5] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,5] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'JOB [9019,5] HAS TERMINATED', 'ACTIVATE JOB [9019,5] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,5] STATE NOTIFY COMPLETED PRI 4', 'state:base:check_job_completed job [9019,5] is terminated (1 vs 1 [NORMALLY TERMINATED])'] Log for succeeded job: [9019,13] ['ACTIVATE JOB [9019,13] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,13] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,13] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,13] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,13] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,13] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,13] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,13] STATE ALLOCATION COMPLETE PRI 4', 'state:base:check_job_completed job [9019,13] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,13] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'ACTIVATE JOB [9019,13] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,13] STATE PENDING DAEMON LAUNCH PRI 4', 'ACTIVATE JOB [9019,13] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,13] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,13] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,13] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,13] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,13] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,13] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,13] STATE MAP COMPLETE PRI 4', 'state:base:check_job_completed job [9019,13] is not terminated (0:1)', 'state:base:check_job_completed job [9019,13] is not terminated (0:1)', 'state:base:check_job_completed job [9019,13] is not terminated (0:1)', 'state:base:check_job_completed job [9019,13] is not terminated (0:1)', 'ACTIVATE JOB [9019,13] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,13] STATE PENDING FINAL SYSTEM PREP PRI 4', 'ACTIVATE JOB [9019,13] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,13] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,13] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,13] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,13] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,13] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,13] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'state:base:check_job_completed job [9019,13] is not terminated (0:1)', 'ACTIVATE JOB [9019,13] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,13] STATE NORMALLY TERMINATED PRI 4', 'state:base:check_job_completed job [9019,13] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,13] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,13] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'JOB [9019,13] HAS TERMINATED', 'ACTIVATE JOB [9019,13] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,13] STATE NOTIFY COMPLETED PRI 4', 'state:base:check_job_completed job [9019,13] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,13] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,13] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,13] is terminated (1 vs 1 [NORMALLY TERMINATED])'] Log for succeeded job: [9019,1] ['ACTIVATE JOB [9019,1] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,1] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,1] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,1] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,1] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,1] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,1] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,1] STATE ALLOCATION COMPLETE PRI 4', 'ACTIVATE JOB [9019,1] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,1] STATE PENDING DAEMON LAUNCH PRI 4', 'ACTIVATE JOB [9019,1] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,1] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,1] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,1] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,1] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,1] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,1] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,1] STATE MAP COMPLETE PRI 4', 'ACTIVATE JOB [9019,1] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,1] STATE PENDING FINAL SYSTEM PREP PRI 4', 'ACTIVATE JOB [9019,1] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,1] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,1] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,1] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,1] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,1] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,1] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'ACTIVATE JOB [9019,1] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,1] STATE NORMALLY TERMINATED PRI 4', 'state:base:check_job_completed job [9019,1] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'JOB [9019,1] HAS TERMINATED', 'ACTIVATE JOB [9019,1] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,1] STATE NOTIFY COMPLETED PRI 4'] Log for succeeded job: [9019,15] ['ACTIVATE JOB [9019,15] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,15] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,15] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,15] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,15] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,15] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,15] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,15] STATE ALLOCATION COMPLETE PRI 4', 'state:base:check_job_completed job [9019,15] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,15] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'ACTIVATE JOB [9019,15] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,15] STATE PENDING DAEMON LAUNCH PRI 4', 'ACTIVATE JOB [9019,15] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,15] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,15] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,15] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,15] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,15] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,15] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,15] STATE MAP COMPLETE PRI 4', 'state:base:check_job_completed job [9019,15] is not terminated (0:1)', 'state:base:check_job_completed job [9019,15] is not terminated (0:1)', 'state:base:check_job_completed job [9019,15] is not terminated (0:1)', 'state:base:check_job_completed job [9019,15] is not terminated (0:1)', 'ACTIVATE JOB [9019,15] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,15] STATE PENDING FINAL SYSTEM PREP PRI 4', 'ACTIVATE JOB [9019,15] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,15] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,15] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,15] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,15] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,15] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,15] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'state:base:check_job_completed job [9019,15] is not terminated (0:1)', 'ACTIVATE JOB [9019,15] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,15] STATE NORMALLY TERMINATED PRI 4', 'state:base:check_job_completed job [9019,15] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'JOB [9019,15] HAS TERMINATED', 'ACTIVATE JOB [9019,15] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,15] STATE NOTIFY COMPLETED PRI 4', 'state:base:check_job_completed job [9019,15] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,15] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,15] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,15] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,15] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,15] is terminated (1 vs 1 [NORMALLY TERMINATED])'] Log for succeeded job: [9019,14] ['ACTIVATE JOB [9019,14] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,14] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,14] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,14] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,14] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,14] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,14] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,14] STATE ALLOCATION COMPLETE PRI 4', 'state:base:check_job_completed job [9019,14] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,14] is terminated (0 vs 0 [NORMALLY TERMINATED])', 'ACTIVATE JOB [9019,14] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,14] STATE PENDING DAEMON LAUNCH PRI 4', 'ACTIVATE JOB [9019,14] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,14] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,14] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,14] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,14] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,14] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,14] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,14] STATE MAP COMPLETE PRI 4', 'state:base:check_job_completed job [9019,14] is not terminated (0:1)', 'state:base:check_job_completed job [9019,14] is not terminated (0:1)', 'state:base:check_job_completed job [9019,14] is not terminated (0:1)', 'state:base:check_job_completed job [9019,14] is not terminated (0:1)', 'ACTIVATE JOB [9019,14] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,14] STATE PENDING FINAL SYSTEM PREP PRI 4', 'ACTIVATE JOB [9019,14] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,14] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,14] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,14] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,14] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,14] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,14] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'state:base:check_job_completed job [9019,14] is not terminated (0:1)', 'ACTIVATE JOB [9019,14] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:556', 'ACTIVATING JOB [9019,14] STATE NORMALLY TERMINATED PRI 4', 'state:base:check_job_completed job [9019,14] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,14] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'JOB [9019,14] HAS TERMINATED', 'ACTIVATE JOB [9019,14] STATE NOTIFY COMPLETED AT ../../../../../orte/mca/state/dvm/state_dvm.c:416', 'ACTIVATING JOB [9019,14] STATE NOTIFY COMPLETED PRI 4', 'state:base:check_job_completed job [9019,14] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,14] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,14] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,14] is terminated (1 vs 1 [NORMALLY TERMINATED])', 'state:base:check_job_completed job [9019,14] is terminated (1 vs 1 [NORMALLY TERMINATED])'] Log for failed job: [9019,3] ['ACTIVATE JOB [9019,3] STATE PENDING INIT AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:888', 'ACTIVATING JOB [9019,3] STATE PENDING INIT PRI 4', 'ACTIVATE JOB [9019,3] STATE INIT_COMPLETE AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:311', 'ACTIVATING JOB [9019,3] STATE INIT_COMPLETE PRI 4', 'ACTIVATE JOB [9019,3] STATE PENDING ALLOCATION AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:322', 'ACTIVATING JOB [9019,3] STATE PENDING ALLOCATION PRI 4', 'ACTIVATE JOB [9019,3] STATE ALLOCATION COMPLETE AT ../../../../orte/mca/ras/base/ras_base_allocate.c:443', 'ACTIVATING JOB [9019,3] STATE ALLOCATION COMPLETE PRI 4', 'ACTIVATE JOB [9019,3] STATE PENDING DAEMON LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:200', 'ACTIVATING JOB [9019,3] STATE PENDING DAEMON LAUNCH PRI 4', 'ACTIVATE JOB [9019,3] STATE ALL DAEMONS REPORTED AT ../../../../../orte/mca/plm/rsh/plm_rsh_module.c:1001', 'ACTIVATING JOB [9019,3] STATE ALL DAEMONS REPORTED PRI 4', 'ACTIVATE JOB [9019,3] STATE VM READY AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:188', 'ACTIVATING JOB [9019,3] STATE VM READY PRI 4', 'ACTIVATE JOB [9019,3] STATE PENDING MAPPING AT ../../../../../orte/mca/state/dvm/state_dvm.c:208', 'ACTIVATING JOB [9019,3] STATE PENDING MAPPING PRI 4', 'ACTIVATE JOB [9019,3] STATE MAP COMPLETE AT ../../../../orte/mca/rmaps/base/rmaps_base_map_job.c:522', 'ACTIVATING JOB [9019,3] STATE MAP COMPLETE PRI 4', 'ACTIVATE JOB [9019,3] STATE PENDING FINAL SYSTEM PREP AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:251', 'ACTIVATING JOB [9019,3] STATE PENDING FINAL SYSTEM PREP PRI 4', 'ACTIVATE JOB [9019,3] STATE PENDING APP LAUNCH AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:463', 'ACTIVATING JOB [9019,3] STATE PENDING APP LAUNCH PRI 4', 'ACTIVATE JOB [9019,3] STATE LOCAL LAUNCH COMPLETE AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1095', 'ACTIVATING JOB [9019,3] STATE LOCAL LAUNCH COMPLETE PRI 4', 'ACTIVATE JOB [9019,3] STATE RUNNING AT ../../../../orte/mca/state/base/state_base_fns.c:488', 'ACTIVATING JOB [9019,3] STATE RUNNING PRI 4', 'ACTIVATE JOB [9019,3] STATE READY FOR DEBUGGERS AT ../../../../orte/mca/plm/base/plm_base_launch_support.c:691', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)', 'state:base:check_job_completed job [9019,3] is not terminated (0:1)'] Log for succeeded proc [[9019,7],0] ['ACTIVATE PROC [[9019,7],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,7],0] STATE RUNNING PRI 4', 'state:base:track_procs called for proc [[9019,7],0] state RUNNING', 'ACTIVATE PROC [[9019,7],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,7],0] STATE IOF COMPLETE PRI 4', 'ACTIVATE PROC [[9019,7],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,7],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,7],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,7],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,7],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,7],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,7],0] state NORMALLY TERMINATED'] Log for succeeded proc [[9019,13],0] ['ACTIVATE PROC [[9019,13],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,13],0] STATE RUNNING PRI 4', 'state:base:track_procs called for proc [[9019,13],0] state RUNNING', 'ACTIVATE PROC [[9019,13],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,13],0] STATE IOF COMPLETE PRI 4', 'ACTIVATE PROC [[9019,13],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,13],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,13],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,13],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,13],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,13],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,13],0] state NORMALLY TERMINATED'] Log for succeeded proc [[9019,11],0] ['ACTIVATE PROC [[9019,11],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,11],0] STATE RUNNING PRI 4', 'state:base:track_procs called for proc [[9019,11],0] state RUNNING', 'ACTIVATE PROC [[9019,11],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,11],0] STATE IOF COMPLETE PRI 4', 'ACTIVATE PROC [[9019,11],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,11],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,11],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,11],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,11],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,11],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,11],0] state NORMALLY TERMINATED'] Log for succeeded proc [[9019,5],0] ['ACTIVATE PROC [[9019,5],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,5],0] STATE RUNNING PRI 4', 'state:base:track_procs called for proc [[9019,5],0] state RUNNING', 'ACTIVATE PROC [[9019,5],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,5],0] STATE IOF COMPLETE PRI 4', 'ACTIVATE PROC [[9019,5],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,5],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,5],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,5],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,5],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,5],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,5],0] state NORMALLY TERMINATED'] Log for succeeded proc [[9019,16],0] ['ACTIVATE PROC [[9019,16],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,16],0] STATE RUNNING PRI 4', 'state:base:track_procs called for proc [[9019,16],0] state RUNNING', 'ACTIVATE PROC [[9019,16],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,16],0] STATE IOF COMPLETE PRI 4', 'ACTIVATE PROC [[9019,16],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,16],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,16],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,16],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,16],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,16],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,16],0] state NORMALLY TERMINATED'] Log for succeeded proc [[9019,10],0] ['ACTIVATE PROC [[9019,10],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,10],0] STATE RUNNING PRI 4', 'state:base:track_procs called for proc [[9019,10],0] state RUNNING', 'ACTIVATE PROC [[9019,10],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,10],0] STATE IOF COMPLETE PRI 4', 'ACTIVATE PROC [[9019,10],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,10],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,10],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,10],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,10],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,10],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,10],0] state NORMALLY TERMINATED'] Log for succeeded proc [[9019,14],0] ['ACTIVATE PROC [[9019,14],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,14],0] STATE RUNNING PRI 4', 'state:base:track_procs called for proc [[9019,14],0] state RUNNING', 'ACTIVATE PROC [[9019,14],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,14],0] STATE IOF COMPLETE PRI 4', 'ACTIVATE PROC [[9019,14],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,14],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,14],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,14],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,14],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,14],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,14],0] state NORMALLY TERMINATED'] Log for succeeded proc [[9019,9],0] ['ACTIVATE PROC [[9019,9],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,9],0] STATE RUNNING PRI 4', 'state:base:track_procs called for proc [[9019,9],0] state RUNNING', 'ACTIVATE PROC [[9019,9],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,9],0] STATE IOF COMPLETE PRI 4', 'ACTIVATE PROC [[9019,9],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,9],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,9],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,9],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,9],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,9],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,9],0] state NORMALLY TERMINATED'] Log for succeeded proc [[9019,12],0] ['ACTIVATE PROC [[9019,12],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,12],0] STATE RUNNING PRI 4', 'state:base:track_procs called for proc [[9019,12],0] state RUNNING', 'ACTIVATE PROC [[9019,12],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,12],0] STATE IOF COMPLETE PRI 4', 'ACTIVATE PROC [[9019,12],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,12],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,12],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,12],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,12],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,12],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,12],0] state NORMALLY TERMINATED'] Log for succeeded proc [[9019,15],0] ['ACTIVATE PROC [[9019,15],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,15],0] STATE RUNNING PRI 4', 'state:base:track_procs called for proc [[9019,15],0] state RUNNING', 'ACTIVATE PROC [[9019,15],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,15],0] STATE IOF COMPLETE PRI 4', 'ACTIVATE PROC [[9019,15],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,15],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,15],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,15],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,15],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,15],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,15],0] state NORMALLY TERMINATED'] Log for succeeded proc [[9019,1],0] ['ACTIVATE PROC [[9019,1],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,1],0] STATE RUNNING PRI 4', 'ACTIVATE PROC [[9019,1],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,1],0] STATE IOF COMPLETE PRI 4', 'state:base:track_procs called for proc [[9019,1],0] state RUNNING', 'ACTIVATE PROC [[9019,1],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,1],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,1],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,1],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,1],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,1],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,1],0] state NORMALLY TERMINATED'] Log for succeeded proc [[9019,8],0] ['ACTIVATE PROC [[9019,8],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,8],0] STATE RUNNING PRI 4', 'ACTIVATE PROC [[9019,8],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,8],0] STATE IOF COMPLETE PRI 4', 'state:base:track_procs called for proc [[9019,8],0] state RUNNING', 'ACTIVATE PROC [[9019,8],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,8],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,8],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,8],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,8],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,8],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,8],0] state NORMALLY TERMINATED'] Log for succeeded proc [[9019,2],0] ['ACTIVATE PROC [[9019,2],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,2],0] STATE RUNNING PRI 4', 'ACTIVATE PROC [[9019,2],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,2],0] STATE IOF COMPLETE PRI 4', 'state:base:track_procs called for proc [[9019,2],0] state RUNNING', 'ACTIVATE PROC [[9019,2],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,2],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,2],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,2],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,2],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,2],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,2],0] state NORMALLY TERMINATED'] Log for succeeded proc [[9019,4],0] ['ACTIVATE PROC [[9019,4],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,4],0] STATE RUNNING PRI 4', 'state:base:track_procs called for proc [[9019,4],0] state RUNNING', 'ACTIVATE PROC [[9019,4],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,4],0] STATE IOF COMPLETE PRI 4', 'ACTIVATE PROC [[9019,4],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,4],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,4],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,4],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,4],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,4],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,4],0] state NORMALLY TERMINATED'] Log for succeeded proc [[9019,6],0] ['ACTIVATE PROC [[9019,6],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,6],0] STATE RUNNING PRI 4', 'state:base:track_procs called for proc [[9019,6],0] state RUNNING', 'ACTIVATE PROC [[9019,6],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,6],0] STATE IOF COMPLETE PRI 4', 'ACTIVATE PROC [[9019,6],0] STATE WAITPID FIRED AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1409', 'ACTIVATING PROC [[9019,6],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,6],0] state IOF COMPLETE', 'state:base:track_procs called for proc [[9019,6],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,6],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,6],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,6],0] state NORMALLY TERMINATED'] Log for failed proc [[9019,3],0] ['ACTIVATE PROC [[9019,3],0] STATE RUNNING AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1064', 'ACTIVATING PROC [[9019,3],0] STATE RUNNING PRI 4', 'ACTIVATE PROC [[9019,3],0] STATE IOF COMPLETE AT ../../../../../orte/mca/iof/hnp/iof_hnp_read.c:280', 'ACTIVATING PROC [[9019,3],0] STATE IOF COMPLETE PRI 4', 'state:base:track_procs called for proc [[9019,3],0] state RUNNING', 'state:base:track_procs called for proc [[9019,3],0] state IOF COMPLETE', 'ACTIVATE PROC [[9019,3],0] STATE KILLED BY INTERNAL COMMAND AT ../../../../orte/mca/odls/base/odls_base_default_fns.c:1611', 'ACTIVATING PROC [[9019,3],0] STATE KILLED BY INTERNAL COMMAND PRI 0', 'ACTIVATE PROC [[9019,3],0] STATE WAITPID FIRED AT ../../../../../orte/mca/errmgr/default_hnp/errmgr_default_hnp.c:641', 'ACTIVATING PROC [[9019,3],0] STATE WAITPID FIRED PRI 4', 'state:base:track_procs called for proc [[9019,3],0] state WAITPID FIRED', 'ACTIVATE PROC [[9019,3],0] STATE NORMALLY TERMINATED AT ../../../../orte/mca/state/base/state_base_fns.c:518', 'ACTIVATING PROC [[9019,3],0] STATE NORMALLY TERMINATED PRI 4', 'state:base:track_procs called for proc [[9019,3],0] state NORMALLY TERMINATED']
