Even though projecting the vector is not the bottleneck I was observing, I've decided to play with it anyway. The results are bizarre to say the least. The following is the total time used by ProjectVector::operator() on a number of processors. This is the code that does the computation, and there is no communication. I have three systems, for a total of 5 vectors which are actually projected:
NP time (sec) --------- 01 28.9 02 15.1 04 15.07 06 15.06 08 15.2 12 7.5 16 7.63 40 3.03 The machine is quad-socket, dual core opteron (8 cores/node). So going from 1 processor to 2 is good scaling. But then it is completely flat until 12 processors (2nodes x 6cores), again flat at 16 (2nodes x 8 cores). Yeah, I checked that the number of elements being processed is decreasing the way you would expect. So there is something magic about 2 cores in a node that maxes out the scalability. I'm guessing there is just a whole lot of memory contention at this point? I'm filling in 10 and 14 right now and will report back. The hardware and application are sufficiently different from Tim's problem that I can't say anything specific about his issue, but this is certainly interesting behavior. -Ben ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Libmesh-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libmesh-devel
