[OMPI users] benchmark - mpi_reduce() called only once but takes long time - proportional to calculation time
Dear users,
I'm running the popular Calculate PI program on a 2 node setting running
ubuntu 8.10 and openmpi1.3.3(with default settings). Password-less ssh
is set up but no cluster management program such as network file system,
network time protocol, resource management, scheduler, etc. The two
nodes are connected though TCP/IP only.
When I tried to benchmark the program, it shows that the time spent on
MPI_Reduce(), is proportional to the Number-of-Intervals (n) used in
calculation. For example, when n = 1,000,000, MPI_Reduce costs 15.65
milliseconds; while n= 1,000,000,000, MPI_Reduce costs 15526 milliseconds.
This confused me - in this Calc-PI program, MPI_Reduce is used only once
- no matter what number of intervals is used, MPI_Reduce is called after
both nodes got the result, to merge the result - just once. So the time
cost by MPI_Reduce (all though it might be slow through TCP/IP
connection) should be somewhat consistent. But obviously it's not what I
saw.
Had anyone have the similar problem before? I'm not sure how
MPI_Reduce() work internally. Does the fact that I don't have network
file system, network time protocol, resource management, scheduler, etc
installed matters?
Below is the program - I did feed "n" to it more than once to warm it up.
#include "mpi.h"
#include
#include
int main(int argc, char *argv[])
{
int numprocs, myid, rc;
double ACCUPI = 3.1415926535897932384626433832795;
double mypi, pi, h, sum, x;
int n, i;
double starttime, endtime;
double time,told,bcasttime,reducetime,comptime,totaltime;
rc = MPI_Init(&argc,&argv);
if (rc != MPI_SUCCESS) {
printf("Error starting MPI program. Terminating.\n");
MPI_Abort(MPI_COMM_WORLD, rc);
}
MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&myid);
while (1) {
if (myid == 0) {
printf("Enter the number of intervals: (0 quits) \n");
scanf("%d",&n);
starttime = MPI_Wtime();
}
time = MPI_Wtime();
MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);
told = time;
time = MPI_Wtime();
bcasttime = time - told;
if (n == 0)
break;
else {
h = 1.0/(double)n;
sum = 0.0;
for (i = myid + 1; i <= n; i += numprocs) {
x = h*((double)i - 0.5);
sum += (4.0/(1.0 + x*x));
}
mypi = sum*h;
told = time;
time = MPI_Wtime();
comptime = time - told;
MPI_Reduce(&mypi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
told = time;
time = MPI_Wtime();
reducetime = time - told;
if (myid == 0) {
totaltime = MPI_Wtime() - starttime;
printf("\nElapsed time (total): %f
milliseconds\n",totaltime*1000);
printf("Elapsed time (Bcast): %f milliseconds
(%5.2f%%)\n",bcasttime*1000,bcasttime*100/totaltime);
printf("Elapsed time (Reduce): %f milliseconds
(%5.2f%%)\n",reducetime*1000,reducetime*100/totaltime);
printf("Elapsed time (Comput): %f milliseconds
(%5.2f%%)\n",comptime*1000,comptime*100/totaltime);
printf("\nApproximated pi is %.16f, Error is %.4e\n", pi,
fabs(pi - ACCUPI));
}
}
}
MPI_Finalize();
}
Re: [OMPI users] benchmark - mpi_reduce() called only once but takes long time - proportional to calculation time
Your processes are probably running asynchronously. You could perhaps
try tracing program execution and look at the timeline. E.g.,
http://www.open-mpi.org/faq/?category=perftools#free-tools . Or, where
you have MPI_Wtime calls, just capture those timestamps on each process
and dump the results at the end of your run. Or, report timings for all
ranks instead of just for rank 0.
Put another way, rank 0 must broadcast n. So, no one starts computation
until they get the Bcast result. Rank 0 probably starts its
computations before anyone else does. So, it gets to the Reduce before
anyone else does, but it can't exit until other ranks have finished
their computations. So, the Reduce time on rank 0 includes some amount
of other ranks' compute times.
Yet another approach is to insert MPI_Barrier calls at each phase of the
program so that the various phases are synchronized. This adds some
overhead to the program, but helps simplify interpretation of the timing
results.
Qing Pang wrote:
I'm running the popular Calculate PI program on a 2 node setting
running ubuntu 8.10 and openmpi1.3.3(with default settings).
Password-less ssh is set up but no cluster management program such as
network file system, network time protocol, resource management,
scheduler, etc. The two nodes are connected though TCP/IP only.
When I tried to benchmark the program, it shows that the time spent on
MPI_Reduce(), is proportional to the Number-of-Intervals (n) used in
calculation. For example, when n = 1,000,000, MPI_Reduce costs 15.65
milliseconds; while n= 1,000,000,000, MPI_Reduce costs 15526
milliseconds.
This confused me - in this Calc-PI program, MPI_Reduce is used only
once - no matter what number of intervals is used, MPI_Reduce is
called after both nodes got the result, to merge the result - just
once. So the time cost by MPI_Reduce (all though it might be slow
through TCP/IP connection) should be somewhat consistent. But
obviously it's not what I saw.
Had anyone have the similar problem before? I'm not sure how
MPI_Reduce() work internally. Does the fact that I don't have network
file system, network time protocol, resource management, scheduler,
etc installed matters?
Below is the program - I did feed "n" to it more than once to warm it up.
#include "mpi.h"
#include
#include
int main(int argc, char *argv[]) { int numprocs, myid, rc;
double ACCUPI = 3.1415926535897932384626433832795;
double mypi, pi, h, sum, x;
int n, i;
double starttime, endtime;
double time,told,bcasttime,reducetime,comptime,totaltime;
rc = MPI_Init(&argc,&argv);
if (rc != MPI_SUCCESS) {
printf("Error starting MPI program. Terminating.\n");
MPI_Abort(MPI_COMM_WORLD, rc);
}
MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&myid);
while (1) {
if (myid == 0) {
printf("Enter the number of intervals: (0 quits) \n");
scanf("%d",&n);
starttime = MPI_Wtime();
}
time = MPI_Wtime();
MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);
told = time;
time = MPI_Wtime();
bcasttime = time - told;
if (n == 0)
break;
else {
h = 1.0/(double)n;
sum = 0.0;
for (i = myid + 1; i <= n; i += numprocs) {
x = h*((double)i - 0.5);
sum += (4.0/(1.0 + x*x));
}
mypi = sum*h;
told = time;
time = MPI_Wtime();
comptime = time - told;
MPI_Reduce(&mypi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0,
MPI_COMM_WORLD);
told = time;
time = MPI_Wtime();
reducetime = time - told;
if (myid == 0) {
totaltime = MPI_Wtime() - starttime;
printf("\nElapsed time (total): %f
milliseconds\n",totaltime*1000);
printf("Elapsed time (Bcast): %f milliseconds
(%5.2f%%)\n",bcasttime*1000,bcasttime*100/totaltime);
printf("Elapsed time (Reduce): %f milliseconds
(%5.2f%%)\n",reducetime*1000,reducetime*100/totaltime);
printf("Elapsed time (Comput): %f milliseconds
(%5.2f%%)\n",comptime*1000,comptime*100/totaltime);
printf("\nApproximated pi is %.16f, Error is %.4e\n", pi,
fabs(pi - ACCUPI));
}
}
}
MPI_Finalize(); }
Re: [OMPI users] benchmark - mpi_reduce() called only once but takes long time - proportional to calculation time
Thank you so much! It is a synchronization issue. In my case, one node
actually run slower than the other node. Adding MPE_Barrier() helps to
straight things out.
Thank you for your help!
Eugene Loh wrote:
Your processes are probably running asynchronously. You could perhaps
try tracing program execution and look at the timeline. E.g.,
http://www.open-mpi.org/faq/?category=perftools#free-tools . Or,
where you have MPI_Wtime calls, just capture those timestamps on each
process and dump the results at the end of your run. Or, report
timings for all ranks instead of just for rank 0.
Put another way, rank 0 must broadcast n. So, no one starts
computation until they get the Bcast result. Rank 0 probably starts
its computations before anyone else does. So, it gets to the Reduce
before anyone else does, but it can't exit until other ranks have
finished their computations. So, the Reduce time on rank 0 includes
some amount of other ranks' compute times.
Yet another approach is to insert MPI_Barrier calls at each phase of
the program so that the various phases are synchronized. This adds
some overhead to the program, but helps simplify interpretation of the
timing results.
Qing Pang wrote:
I'm running the popular Calculate PI program on a 2 node setting
running ubuntu 8.10 and openmpi1.3.3(with default settings).
Password-less ssh is set up but no cluster management program such as
network file system, network time protocol, resource management,
scheduler, etc. The two nodes are connected though TCP/IP only.
When I tried to benchmark the program, it shows that the time spent
on MPI_Reduce(), is proportional to the Number-of-Intervals (n) used
in calculation. For example, when n = 1,000,000, MPI_Reduce costs
15.65 milliseconds; while n= 1,000,000,000, MPI_Reduce costs 15526
milliseconds.
This confused me - in this Calc-PI program, MPI_Reduce is used only
once - no matter what number of intervals is used, MPI_Reduce is
called after both nodes got the result, to merge the result - just
once. So the time cost by MPI_Reduce (all though it might be slow
through TCP/IP connection) should be somewhat consistent. But
obviously it's not what I saw.
Had anyone have the similar problem before? I'm not sure how
MPI_Reduce() work internally. Does the fact that I don't have network
file system, network time protocol, resource management, scheduler,
etc installed matters?
Below is the program - I did feed "n" to it more than once to warm it
up.
#include "mpi.h"
#include
#include
int main(int argc, char *argv[]) { int numprocs, myid, rc;
double ACCUPI = 3.1415926535897932384626433832795;
double mypi, pi, h, sum, x;
int n, i;
double starttime, endtime;
double time,told,bcasttime,reducetime,comptime,totaltime;
rc = MPI_Init(&argc,&argv);
if (rc != MPI_SUCCESS) {
printf("Error starting MPI program. Terminating.\n");
MPI_Abort(MPI_COMM_WORLD, rc);
}
MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&myid);
while (1) {
if (myid == 0) {
printf("Enter the number of intervals: (0 quits) \n");
scanf("%d",&n);
starttime = MPI_Wtime();
}
time = MPI_Wtime();
MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);
told = time;
time = MPI_Wtime();
bcasttime = time - told;
if (n == 0)
break;
else {
h = 1.0/(double)n;
sum = 0.0;
for (i = myid + 1; i <= n; i += numprocs) {
x = h*((double)i - 0.5);
sum += (4.0/(1.0 + x*x));
}
mypi = sum*h;
told = time;
time = MPI_Wtime();
comptime = time - told;
MPI_Reduce(&mypi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0,
MPI_COMM_WORLD);
told = time;
time = MPI_Wtime();
reducetime = time - told;
if (myid == 0) {
totaltime = MPI_Wtime() - starttime;
printf("\nElapsed time (total): %f
milliseconds\n",totaltime*1000);
printf("Elapsed time (Bcast): %f milliseconds
(%5.2f%%)\n",bcasttime*1000,bcasttime*100/totaltime);
printf("Elapsed time (Reduce): %f milliseconds
(%5.2f%%)\n",reducetime*1000,reducetime*100/totaltime);
printf("Elapsed time (Comput): %f milliseconds
(%5.2f%%)\n",comptime*1000,comptime*100/totaltime);
printf("\nApproximated pi is %.16f, Error is %.4e\n", pi,
fabs(pi - ACCUPI));
}
}
}
MPI_Finalize(); }
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
