Begin forwarded message:
> From: John Shalf <jshalf at lbl.gov> > Date: June 19, 2011 5:15:49 PM CDT > To: Barry Smith <bsmith at mcs.anl.gov> > Cc: Nathan Wichmann <wichmann at cray.com>, Lois Curfman McInnes <curfman at > mcs.anl.gov>, Satish Balay <balay at mcs.anl.gov>, Alice Koniges <aekoniges > at lbl.gov>, Robert Preissl <rpreissl at lbl.gov>, Erich Strohmaier > <EStrohmaier at lbl.gov>, Stephane Ethier <ethier at pppl.gov> > Subject: Re: Poisson step in GTS > > > On Jun 19, 2011, at 9:44 PM, Barry Smith wrote: >> On Jun 19, 2011, at 5:34 AM, John Shalf wrote: >>> Hi Barry, >>> here is the stream benchmark results that Hongzhang Shan collected on >>> Hopper for Nick's COE studies. The red curve shows performance when you >>> run stream when all of the data ends up mapped to a single memory >>> controller. The blue curve shows the case when you correctly map data >>> using first-touch so that the stream benchmark accesses data on its local >>> memory controller (the correct NUMA mapping). >> >> How does one "correctly map data using first-touch"? (Reference ok). > > The AMD nodes (and even the Intel Nehalem nodes) have memory controllers > integrated onto the processor chips. The processor chips are integrated > together into a node using HyperTransport for the AMD chips or QPI, which > happens to be slower per link than the memory bandwidth of the local memory > controllers on each these chips. Consequently, the bandwidth of accessing > memory using the memory controller that is co-located on the die with the > CPUs is much lower latency and higher bandwidth than accessing the memory > controller on one of the other dies. > > So you need to have some way of identifying which memory controller should > "own" a piece of memory so that you can keep it closer to the processors that > will primarily be using it. The "first touch" memory affinity policy says > that a memory page gets mapped to the memory controller that is *closest* to > the first processor core to write a value to that page. So you can malloc() > data at any time, but the pages get assigned to memory controllers base on > the first processor to "touch" that memory page. If you touch the data (and > thereby assign it to memory controllers) in a different layout than you > actually use it, then most accesses will be non-local, and therefore will be > very slow. If you touch data using the processor that will primarily be > accessing the data later on, then it will get mapped to a local memory > controller and will therefore be much faster. So you have to be very > carefully about how you first touch data to ensure good memory/stream > performance. > > You can ref. the NERSC FAQ on the "first touch" principle. > http://www.nersc.gov/users/computational-systems/hopper/getting-started/multi-core-faq/ > >>> The bottom line is that it is essential that data is touched first on the >>> memory controller that is nearest the OpenMP processes that will be >>> accessing it (otherwise memory bandwidth will tank). This should occur >>> naturally if you configure as 4 NUMA nodes with 6 threads each, as per >>> Nathan's suggestion. >> >> How does one "configure as 4 NUMA nodes with 6 threads each"? Do you mean 4 >> MPI processes (each with 6 threads or something different?) > > That is correct. The Cray XE6 node (the dual-socket AMD magnycours) has a > total of 4 dies (and hence 4 NUMA domains). Cray refers to these dies (each > of which has its own local memory controller) as a NUMA-node to distinguish > it from the full node that contains for of these dies. Within a NUMA node, > there a no NUMA issues. > > So Cray refers to these dies (these sub-sections of a node where there are no > NUMA issues) as numa_nodes. You can use 'aprun' to launch tasks so that you > get one task per numa_node and the threads within that numa_node will not > have to worry about the first touch stuff we talked about above. For Hopper, > that is 4 numa_nodes per node, and 6 OpenMP threads per numa_node. > > e.g. > aprun -S threads_per_numa_node=6 -sn numa_nodes_per_node=4 > >>> If we want to be more aggressive and use 24-way threaded parallelism per >>> node, then extra care must be taken to ensure the memory affinity is not >>> screwed up. >> >> BTW: What is an "OpenMP thread" mapped to on the Cray systems? A pthread? >> Some other kind of thread? > > I'm not sure what you mean here. OpenMP is directives for threading. So an > OpenMP "thread" is just how many threads you assign to each MPI task (with > OpenMP operating within each MPI task). > > -john >
