Thanks for the reply. I tried reducing the number of loops from 5000, I went as low at 7, and still measured times of 1ms per communication. Below 7 loops I measured a time of zero, but MPI_Wtime() only has a resolution of 4ms.
I wasn't sure what you meant by handling the return status. I changed the code to check the returned error value, but find no errors. And I find putting MPI_Barrier(MPI_COMM_WORLD) after every communication reduces the time per communication from 1ms to 70microseconds. ---------- Forwarded Message ---------- Subject: [Beowulf] Re: Why is communication so expensive for very small messages Date: Tue, 24 Apr 2007 12:30:03 -0700 From: "David Mathog" <[EMAIL PROTECTED]> To: [email protected] Jonathan Boyle <[EMAIL PROTECTED]> wrote: > I apologise if this is a naive question, but I'm new to this world of > beowulfs. > > I'm using C++/mpi, to get a feel for communication costs I ran tests using > mpptest and my own programs. > > For 2 processor blocking calls, mpptest indicates a latency of about 30 > microseconds. > > However when I measure communication times in my own program using a loop as > follows.... > > MPI_Barrier(MPI_COMM_WORLD); > start = MPI_Wtime(); > for (unsigned t=1; t<=5000; t++) > { > if (my_rank==0) > { > MPI_Send(data, size, MPI_INT, 1, tag, MPI_COMM_WORLD); > } > else > { > MPI_Recv(data, size, MPI_INT, 0, tag, MPI_COMM_WORLD, &status); > } > } > end = MPI_Wtime(); > > for size>=4, I get a latency of about 30 microseconds as expected, however for > size<4, communication costs increase massively, and latency now appears to be > 1ms! Odd. Could be a couple of things going on. 1. By pure bad luck dropping size 4 -> 3 (for instance) may be the point at which your network saturates. That is, the slightly smaller packets may be generated slightly faster so that you hit a bottleneck (max packets per second inbound would be my guess, I've hit that limit before) and so you see a nonlinear response. You can test this by dropping the 5000 to 100 or so, so no matter how fast they go, they don't max anything out. 2. Could be a bug in the mpi implementation you are using. I've never written anything for MPI in C++, is the return status being handled somewhere? I don't see it in this code fragment but C++ is notorious for putting side effects in another section of code, far far away. If at all possible the status values should be checked. Sooner or later one of these functions is going to come back with a bad status and your program should handle that. Regards, David Mathog [EMAIL PROTECTED] Manager, Sequence Analysis Facility, Biology Division, Caltech _______________________________________________ Beowulf mailing list, [email protected] To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
