Hi,
My version of OpenMPI is 1.5.3.I have attached a simple toy code. Which is
actually modification of example given on the web
http://osl.iu.edu/research/ft/ompi-cr/examples.php .Mainly, i introduced some
communication between processes and every process writes its separate
checkpoint file.Here is my command line for running the job.
>> mpirun -np 48 -npernode 6 -bind-to-core -bycore -am ft-enable-cr
--mca crs_self_prefix my_personal ./SELF_CR 5000
Also i have attached another file containing my MCA options from 'ompi_info'.
In this toy-code, problem is not too severe, so i used 48 or even 96 processes
and many checkpoints to make problem appear. But i my actual code, perhaps due
to more MPI calls, sometimes problem occur even within one node with only few
(2-5) processes as well.
Hope to hear from you.Kind regards,Faisal Shahzad
> Date: Wed, 31 Aug 2011 11:35:55 -0400
> From: jjhur...@open-mpi.org
> To: us...@open-mpi.org
> Subject: Re: [OMPI users] Question regarding SELF-checkpointing
>
> That seems like a bug to me.
>
> What version of Open MPI are you using? How have you setup the C/R
> functionality (what MCA options do you have set, what command line
> options are you using)? Can you send a small reproducing application
> that we can test against?
>
> That should help us focus in on the problem a bit.
>
> -- Josh
>
> On Wed, Aug 31, 2011 at 6:36 AM, Faisal Shahzad <itsfa...@hotmail.com> wrote:
> > Dear Group,
> > I have a mpi-program in which every process is communicating with its
> > neighbors. When SELF-checkpointing, every process writes to a separate file.
> > Problem is that sometimes after making a checkpoint, program does not
> > continue again. Having more number of processes makes this problem severe.
> > With just 1 process (no communication), SEFL-checkpointing works normally
> > with no problem.
> > I have tried different '--mca btl' parameters (openib,tcp,sm,self), but
> > problem persists.
> > I would very much appreciate your support regarding it.
> > Kind regards,
> > Faisal
> > _______________________________________________
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
>
>
>
> --
> Joshua Hursey
> Postdoctoral Research Associate
> Oak Ridge National Laboratory
> http://users.nccs.gov/~jjhursey
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
#include <mpi.h>
#include <signal.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#define LIMIT 50
/************************
* Function Declarations
************************/
void signal_handler(int sig);
/* Default OPAL crs self callback functions */
int opal_crs_self_user_checkpoint(char **restart_cmd);
int opal_crs_self_user_continue(void);
int opal_crs_self_user_restart(void);
/* OPAL crs self callback functions */
int my_personal_checkpoint(char **restart_cmd);
int my_personal_continue(void);
int my_personal_restart(void);
/*******************
* Global Variables and arrays
*******************/
int myrank, numprocs;
double *a, *b, *c, *d;
int am_done = 1;
const int N=1000000; // Defines size of arrays
int inr_loop_counter = 0;
int out_loop_counter = 0;
int current_step = 0;
char ckpt_file[128];
char restart_path[128] = "my-restart-path../SELF_CR";
int main(int argc, char** argv){
MPI_Init(&argc, &argv);
int input_time;
if( argc != 2 ) {
input_time = 200;
}
else{
input_time = atoi(argv[1]);
}
/* So we can exit cleanly */
signal(SIGINT, signal_handler);
signal(SIGTERM, signal_handler);
// printf("Received Signal \n");
MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
MPI_Status status[4];
MPI_Request request[4];
a = (double *) malloc( N*sizeof(double) );
b = (double *) malloc( N*sizeof(double) );
c = (double *) malloc( N*sizeof(double) );
d = (double *) malloc( N*sizeof(double) );
int i=0;
for(i=0; i<N ; ++i)
{
a[i] = 1.;
b[i] = 2.;
c[i] = 3.;
d[i] = 4.;
}
// Neighbor's rank calculation
int my_neigh_right, my_neigh_left;
my_neigh_right = myrank+1;
my_neigh_left = myrank-1;
if(myrank==numprocs-1) // for extreme last process
my_neigh_right = 0;
if(myrank==0) // for first process
my_neigh_left = numprocs-1;
if(myrank==0)
printf("Loop starting...\n");
// ==========================================================
// ======================KERNEL LOOP========================= /
for( ; out_loop_counter < input_time ; ++out_loop_counter){
if(myrank==0 && out_loop_counter%50==0){
printf("Outer loop counter : %d\n", out_loop_counter);
}
for(inr_loop_counter = 0; inr_loop_counter < N ; ++inr_loop_counter)
{
a[inr_loop_counter] = b[inr_loop_counter]+c[inr_loop_counter]*d[inr_loop_counter]; // Some dummy calculation
}
// communication between neighboring processes
MPI_Irecv(&b[0],10000, MPI_DOUBLE, my_neigh_left, 0, MPI_COMM_WORLD, &request[0]);
MPI_Isend(&a[0],10000, MPI_DOUBLE, my_neigh_right, 0, MPI_COMM_WORLD, &request[1]);
MPI_Isend(&c[0],10000, MPI_DOUBLE, my_neigh_left, 1, MPI_COMM_WORLD, &request[2]);
MPI_Irecv(&d[0],10000, MPI_DOUBLE, my_neigh_right, 1, MPI_COMM_WORLD, &request[3]);
MPI_Waitall(4, request, status);
MPI_Barrier(MPI_COMM_WORLD);
}
free(a);free(b);free(c);free(d);
MPI_Finalize();
return 0;
}
void signal_handler(int sig) {
printf("Received Signal %d\n", sig);
am_done = 0;
}
/* OPAL crs self callbacks for checkpoint */
int opal_crs_self_user_checkpoint(char **restart_cmd) {
printf("opal_crs_self_user_checkpoint callback...\n");
my_personal_checkpoint(restart_cmd);
return 0;
}
int opal_crs_self_user_continue(void) {
printf("opal_crs_self_user_continue callback...\n");
my_personal_continue();
return 0;
}
int opal_crs_self_user_restart(void) {
printf("opal_crs_self_user_restart callback...\n");
my_personal_restart();
return 0;
}
/* OPAL crs self callback for checkpoint */
int my_personal_checkpoint(char **restart_cmd) {
FILE *fp;
*restart_cmd = NULL;
printf("my_personal_checkpoint callback...\n");
strcpy(ckpt_file,"CP_file_");
char temp_char[64];
sprintf(temp_char, "%d", myrank);
strcat(temp_char, ".ckpt");
strcpy(ckpt_file, strcat(ckpt_file, temp_char));
/*
* Open our checkpoint file
*/
if( NULL == (fp = fopen(ckpt_file, "w")) ) {
fprintf(stderr, "Error: Unable to open file (%s)\n", ckpt_file);
return 0;
}
/*
* Save the process state
*/
int i=0;
for(i = 0; i< N ; ++i){
fprintf(fp, "%lf\n", a[i]);
}
i =0;
for(i = 0; i< N ; ++i){
fprintf(fp, "%lf\n", b[i]);
}
i =0;
/*
* Close the checkpoint file
*/
fclose(fp);
/*
* Figure out the restart command
*/
asprintf(restart_cmd, "%s", strdup(restart_path));
printf("my_personal_checkpoint callback finished...: %d\n",myrank);
return 0;
}
int my_personal_continue() {
printf("my_personal_continue callback...\n");
/* Don't need to do anything here since we are in the
* state that we want to be in already.
*/
return 0;
}
int my_personal_restart() {
printf("my_personal_restart callback...\n");
return 0;
}C bindings: yes
C++ bindings: yes
Fortran77 bindings: yes (all)
Fortran90 bindings: yes
Fortran90 bindings size: small
C compiler: icc
C compiler absolute: /apps/intel/Compiler/11.1/075/bin/intel64/icc
C compiler family name: INTEL
C compiler version: 1110.20101201
C++ compiler: icpc
C++ compiler absolute: /apps/intel/Compiler/11.1/075/bin/intel64/icpc
Fortran77 compiler: ifort
Fortran77 compiler abs: /apps/intel/Compiler/11.1/075/bin/intel64/ifort
Fortran90 compiler: ifort
Fortran90 compiler abs: /apps/intel/Compiler/11.1/075/bin/intel64/ifort
C profiling: yes
C++ profiling: yes
Fortran77 profiling: yes
Fortran90 profiling: yes
C++ exceptions: no
Thread support: posix (mpi: yes, progress: no)
Sparse Groups: no
Internal debug support: no
MPI interface warnings: no
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
libltdl support: yes
Heterogeneous support: no
mpirun default --prefix: no
MPI I/O support: yes
MPI_WTIME support: gettimeofday
Symbol vis. support: yes
MPI extensions: affinity example
FT Checkpoint support: yes (checkpoint thread: yes)
MPI_MAX_PROCESSOR_NAME: 256
MPI_MAX_ERROR_STRING: 256
MPI_MAX_OBJECT_NAME: 64
MPI_MAX_INFO_KEY: 36
MPI_MAX_INFO_VAL: 256
MPI_MAX_PORT_NAME: 1024
MPI_MAX_DATAREP_STRING: 128
MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.5.3)
MCA memory: linux (MCA v2.0, API v2.0, Component v1.5.3)
MCA paffinity: hwloc (MCA v2.0, API v2.0, Component v1.5.3)
MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.5.3)
MCA carto: file (MCA v2.0, API v2.0, Component v1.5.3)
MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.5.3)
MCA timer: linux (MCA v2.0, API v2.0, Component v1.5.3)
MCA installdirs: env (MCA v2.0, API v2.0, Component v1.5.3)
MCA installdirs: config (MCA v2.0, API v2.0, Component v1.5.3)
MCA crs: none (MCA v2.0, API v2.0, Component v1.5.3)
MCA dpm: orte (MCA v2.0, API v2.0, Component v1.5.3)
MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.5.3)
MCA allocator: basic (MCA v2.0, API v2.0, Component v1.5.3)
MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.5.3)
MCA coll: basic (MCA v2.0, API v2.0, Component v1.5.3)
MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.5.3)
MCA coll: inter (MCA v2.0, API v2.0, Component v1.5.3)
MCA coll: self (MCA v2.0, API v2.0, Component v1.5.3)
MCA coll: sm (MCA v2.0, API v2.0, Component v1.5.3)
MCA coll: sync (MCA v2.0, API v2.0, Component v1.5.3)
MCA coll: tuned (MCA v2.0, API v2.0, Component v1.5.3)
MCA io: romio (MCA v2.0, API v2.0, Component v1.5.3)
MCA mpool: fake (MCA v2.0, API v2.0, Component v1.5.3)
MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.5.3)
MCA mpool: sm (MCA v2.0, API v2.0, Component v1.5.3)
MCA pml: bfo (MCA v2.0, API v2.0, Component v1.5.3)
MCA pml: crcpw (MCA v2.0, API v2.0, Component v1.5.3)
MCA pml: csum (MCA v2.0, API v2.0, Component v1.5.3)
MCA pml: ob1 (MCA v2.0, API v2.0, Component v1.5.3)
MCA pml: v (MCA v2.0, API v2.0, Component v1.5.3)
MCA bml: r2 (MCA v2.0, API v2.0, Component v1.5.3)
MCA rcache: vma (MCA v2.0, API v2.0, Component v1.5.3)
MCA btl: ofud (MCA v2.0, API v2.0, Component v1.5.3)
MCA btl: openib (MCA v2.0, API v2.0, Component v1.5.3)
MCA btl: self (MCA v2.0, API v2.0, Component v1.5.3)
MCA btl: sm (MCA v2.0, API v2.0, Component v1.5.3)
MCA btl: tcp (MCA v2.0, API v2.0, Component v1.5.3)
MCA topo: unity (MCA v2.0, API v2.0, Component v1.5.3)
MCA osc: pt2pt (MCA v2.0, API v2.0, Component v1.5.3)
MCA osc: rdma (MCA v2.0, API v2.0, Component v1.5.3)
MCA crcp: bkmrk (MCA v2.0, API v2.0, Component v1.5.3)
MCA iof: hnp (MCA v2.0, API v2.0, Component v1.5.3)
MCA iof: orted (MCA v2.0, API v2.0, Component v1.5.3)
MCA iof: tool (MCA v2.0, API v2.0, Component v1.5.3)
MCA oob: tcp (MCA v2.0, API v2.0, Component v1.5.3)
MCA odls: default (MCA v2.0, API v2.0, Component v1.5.3)
MCA ras: cm (MCA v2.0, API v2.0, Component v1.5.3)
MCA ras: slurm (MCA v2.0, API v2.0, Component v1.5.3)
MCA ras: tm (MCA v2.0, API v2.0, Component v1.5.3)
MCA rmaps: load_balance (MCA v2.0, API v2.0, Component v1.5.3)
MCA rmaps: rank_file (MCA v2.0, API v2.0, Component v1.5.3)
MCA rmaps: resilient (MCA v2.0, API v2.0, Component v1.5.3)
MCA rmaps: round_robin (MCA v2.0, API v2.0, Component v1.5.3)
MCA rmaps: seq (MCA v2.0, API v2.0, Component v1.5.3)
MCA rmaps: topo (MCA v2.0, API v2.0, Component v1.5.3)
MCA rml: ftrm (MCA v2.0, API v2.0, Component v1.5.3)
MCA rml: oob (MCA v2.0, API v2.0, Component v1.5.3)
MCA routed: binomial (MCA v2.0, API v2.0, Component v1.5.3)
MCA routed: cm (MCA v2.0, API v2.0, Component v1.5.3)
MCA routed: direct (MCA v2.0, API v2.0, Component v1.5.3)
MCA routed: linear (MCA v2.0, API v2.0, Component v1.5.3)
MCA routed: radix (MCA v2.0, API v2.0, Component v1.5.3)
MCA routed: slave (MCA v2.0, API v2.0, Component v1.5.3)
MCA plm: rsh (MCA v2.0, API v2.0, Component v1.5.3)
MCA plm: rshd (MCA v2.0, API v2.0, Component v1.5.3)
MCA plm: slurm (MCA v2.0, API v2.0, Component v1.5.3)
MCA plm: tm (MCA v2.0, API v2.0, Component v1.5.3)
MCA snapc: full (MCA v2.0, API v2.0, Component v1.5.3)
MCA filem: rsh (MCA v2.0, API v2.0, Component v1.5.3)
MCA errmgr: default (MCA v2.0, API v2.0, Component v1.5.3)
MCA ess: env (MCA v2.0, API v2.0, Component v1.5.3)
MCA ess: hnp (MCA v2.0, API v2.0, Component v1.5.3)
MCA ess: singleton (MCA v2.0, API v2.0, Component v1.5.3)
MCA ess: slave (MCA v2.0, API v2.0, Component v1.5.3)
MCA ess: slurm (MCA v2.0, API v2.0, Component v1.5.3)
MCA ess: slurmd (MCA v2.0, API v2.0, Component v1.5.3)
MCA ess: tm (MCA v2.0, API v2.0, Component v1.5.3)
MCA ess: tool (MCA v2.0, API v2.0, Component v1.5.3)
MCA grpcomm: bad (MCA v2.0, API v2.0, Component v1.5.3)
MCA grpcomm: basic (MCA v2.0, API v2.0, Component v1.5.3)
MCA grpcomm: hier (MCA v2.0, API v2.0, Component v1.5.3)
MCA notifier: command (MCA v2.0, API v1.0, Component v1.5.3)
MCA notifier: syslog (MCA v2.0, API v1.0, Component v1.5.3)
