I need an account on ranger. ;-)
It is a supermicro board, but I think the topology is the same. Four
hypertransport connections per socket, and some of those connect to the
PCIExpress bus. So that is what you end up with.
I have not messed with numactl, only built mpi with libnuma...
-Ben
----- Original Message -----
From: John Peterson <[EMAIL PROTECTED]>
To: Kirk, Benjamin (JSC-EG)
Cc: [email protected] <[email protected]>
Sent: Tue Sep 09 17:05:15 2008
Subject: Re: [Libmesh-devel] ProjectVector::operator()
On Tue, Sep 9, 2008 at 4:35 PM, Benjamin Kirk <[EMAIL PROTECTED]> wrote:
> Even though projecting the vector is not the bottleneck I was observing,
> I've decided to play with it anyway. The results are bizarre to say the
> least. The following is the total time used by ProjectVector::operator() on
> a number of processors. This is the code that does the computation, and
> there is no communication. I have three systems, for a total of 5 vectors
> which are actually projected:
>
> NP time (sec)
> ---------
> 01 28.9
> 02 15.1
> 04 15.07
> 06 15.06
> 08 15.2
> 12 7.5
> 16 7.63
> 40 3.03
>
> The machine is quad-socket, dual core opteron (8 cores/node).
Are these similar to the ranger nodes, where two of the CPUs are a
little further apart from each other than they are from all the
others? If so, a third thread created on a node may need to go over
two "hops" to access the memory...
If the attachment makes it through, it shows the general idea.
Have you messed around with numactl at all for these nodes? At TACC,
somebody wrote up a script for ranger
(/share/sge/default/pe_scripts/tacc_affinity) to try and get the best
numactl settings for the number of threads requested.
--
John
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Libmesh-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/libmesh-devel
