Hi Bill and Ralph,
well the Linux kernel does all its best to allocate memory on the local
NUMA node if it's available, so it is difficult to convince it to do
something harmful in this sense. I think that a way to test such a
situation would be to start mpi processes on a node in an usual way
-reasonably the processes will be bound to a socket or a core-, wait for
the processes to allocate their working memory, then either migrate the
processes on the other NUMA node (usually ==socket) or migrate its
memory pages, the command-line tools distributed with the numactl
package (numactl or migratepages) can probably allow to perform such a
vandalism; this would put your system into a worst-case scenario from
the NUMA point of view.
In our system, I noticed in the past some strong slowdowns related to
NUMA in parallel processes when a single MPI process doing much more I/O
than the others tended to occupy all the local memory as disk cache,
then the processes on that NUMA node were forced to allocate memory on
the other NUMA node rather than reclaiming cache memory on the local
node. I solved this in a brutal way by cleaning the disk cache regularly
on the computing nodes. In my view this is the only case where (recent)
Linux kernel does not have a NUMA-aware behavior, I wonder whether there
are HPC-optimized patches or something has changed in this direction in
recent kernel development.
Best regards, Davide
Date: Fri, 24 Jul 2015 13:36:55 -0700
From: Ralph Castain <[email protected]>
To: Open MPI Users <[email protected]>
Subject: Re: [OMPI users] NUMA: Non-local memory access and
performance effects on OpenMPI
Hi Bill
You actually can get OMPI to split a process across sockets. Let?s say there
are 4 cores/socket and 2 sockets/node. You could run two procs on the same
node, one split across sockets, by:
mpirun -n 1 ?map-by core:pe=5 ./app : -n 1 ?map-by core:pe=3 ./app
The first proc will run on all cores of the 1st socket plus the 1st core of the
2nd socket. The second proc will run on the remaining 3 cores of the 2nd socket.
HTH
Ralph
On Jul 24, 2015, at 12:48 PM, Lane, William <[email protected]> wrote:
I'm just curious, if we run an OpenMPI job and it makes use of non-local memory
(i.e. memory tied to another socket) what kind of effects are seen on
performance?
How would you go about testing the above? I can't think of any command line
parameter that
would allow one to split an OpenMPI process across sockets.
I'd imagine it would be pretty bad since you can't cache non-local memory
locally,
the fact both the request and data have to flow through an IOH, the local CPU
would
have to compete w/the non-local CPU for access to its own memory and that doing
this
would have to implemented w/some sort of software semaphore locks (which would
add
even more overhead).
Bill L.
IMPORTANT WARNING: This message is intended for the use of the person or entity
to which it is addressed and may contain information that is privileged and
confidential, the disclosure of which is governed by applicable law. If the
reader of this message is not the intended recipient, or the employee or agent
responsible for delivering it to the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this information is
strictly prohibited. Thank you for your cooperation.
_______________________________________________
users mailing list
[email protected] <mailto:[email protected]>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
<http://www.open-mpi.org/mailman/listinfo.cgi/users>
Link to this post: http://www.open-mpi.org/community/lists/users/2015/07/27322.php
<http://www.open-mpi.org/community/lists/users/2015/07/27322.php>
--
============================= Davide Cesari ============================
Dott**(0.5) Davide Cesari
ARPA-Emilia Romagna, Servizio IdroMeteoClima
NWP modelling - Modellistica numerica previsionale
Tel. +39 051525926
========================================================================
