Re: [OMPI users] Processor/core selection/affinity for large shared memory systems
Hi, 1. please, provide #cat /proc/cpu_info 2. see http://www.open-mpi.org/faq/?category=tuning#paffinity-defs. Best regards Lenny.
Re: [OMPI users] Processor/core selection/affinity for large shared memory systems
Terry Frankcombe wrote: > Isn't it up to the OS scheduler what gets run where? I was under the impression that the processor affinity API was designed to let the OS (at least Linux) know how a given task preferred to be bound in terms of the system topology. -- V. Ram v_r_...@fastmail.fm -- http://www.fastmail.fm - Accessible with your email software or over the web
Re: [OMPI users] Processor/core selection/affinity for large shared memory systems
Ralph Castain wrote: > Thanks - yes, that helps. Can you do add --display-map to you cmd > line? That will tell us what mpirun thinks it is doing. The output from display map is below. Note that I've sanitized a few items, but nothing relevant to this: [granite:29685] Map for job: 1 Generated by mapping mode: byslot Starting vpid: 0Vpid range: 16 Num app_contexts: 1 Data for app_context: index 0 app: /path/to/executable Num procs: 16 Argv[0]: /path/to/executable Env[0]: OMPI_MCA_rmaps_base_display_map=1 Env[1]: OMPI_MCA_orte_precondition_transports=e16b0004a956445e-0515b892592a4a02 Env[2]: OMPI_MCA_rds=proxy Env[3]: OMPI_MCA_ras=proxy Env[4]: OMPI_MCA_rmaps=proxy Env[5]: OMPI_MCA_pls=proxy Env[6]: OMPI_MCA_rmgr=proxy Working dir: /home/user/case (user: 0) Num maps: 0 Num elements in nodes list: 1 Mapped node: Cell: 0 Nodename: graniteLaunch id: -1 Username: NULL Daemon name: Data type: ORTE_PROCESS_NAMEData Value: NULL Oversubscribed: TrueNum elements in procs list: 16 Mapped proc: Proc Name: Data type: ORTE_PROCESS_NAMEData Value: [0,1,0] Proc Rank: 0Proc PID: 0 App_context index: 0 Mapped proc: Proc Name: Data type: ORTE_PROCESS_NAMEData Value: [0,1,1] Proc Rank: 1Proc PID: 0 App_context index: 0 Mapped proc: Proc Name: Data type: ORTE_PROCESS_NAMEData Value: [0,1,2] Proc Rank: 2Proc PID: 0 App_context index: 0 Mapped proc: Proc Name: Data type: ORTE_PROCESS_NAMEData Value: [0,1,3] Proc Rank: 3Proc PID: 0 App_context index: 0 Mapped proc: Proc Name: Data type: ORTE_PROCESS_NAMEData Value: [0,1,4] Proc Rank: 4Proc PID: 0 App_context index: 0 Mapped proc: Proc Name: Data type: ORTE_PROCESS_NAMEData Value: [0,1,5] Proc Rank: 5Proc PID: 0 App_context index: 0 Mapped proc: Proc Name: Data type: ORTE_PROCESS_NAMEData Value: [0,1,6] Proc Rank: 6Proc PID: 0 App_context index: 0 Mapped proc: Proc Name: Data type: ORTE_PROCESS_NAMEData Value: [0,1,7] Proc Rank: 7Proc PID: 0 App_context index: 0 Mapped proc: Proc Name: Data type: ORTE_PROCESS_NAMEData Value: [0,1,8] Proc Rank: 8Proc PID: 0 App_context index: 0 Mapped proc: Proc Name: Data type: ORTE_PROCESS_NAMEData Value: [0,1,9] Proc Rank: 9Proc PID: 0 App_context index: 0 Mapped proc: Proc Name: Data type: ORTE_PROCESS_NAMEData Value: [0,1,10] Proc Rank: 10 Proc PID: 0 App_context index: 0 Mapped proc: Proc Name: Data type: ORTE_PROCESS_NAMEData Value: [0,1,11] Proc Rank: 11 Proc PID: 0 App_context index: 0 Mapped proc: Proc Name: Data type: ORTE_PROCESS_NAMEData Value: [0,1,12] Proc Rank: 12 Proc PID: 0 App_context index: 0 Mapped proc: Proc Name: Data type: ORTE_PROCESS_NAMEData Value: [0,1,13] Proc Rank: 13
Re: [OMPI users] Processor/core selection/affinity for large shared memory systems
Isn't it up to the OS scheduler what gets run where? > We have an 8-way, 32-core AMD processor machine (single system image) > and are at present running OpenMPI 1.2.8 . Jobs are launched locally on > the machine itself. As far as I can see, there doesn't seem to be any > way to tell OpenMPI to launch the MPI processes on adjacent cores. > Presumably such functionality is technically possible via PLPA. Is > there in fact a way to specify such a thing with 1.2.8, and if not, will > 1.3 support these kinds arguments? > > Thank you. > -- > V. Ram > v_r_...@fastmail.fm > > -- > http://www.fastmail.fm - Or how I learned to stop worrying and > love email again > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
Re: [OMPI users] Processor/core selection/affinity for large shared memory systems
Ralph H. Castain wrote: > I confess to confusion. OpenMPI will by default map your processes in > a round-robin fashion based on process slot. If you are in a resource > managed environment (e.g., TM or SLURM), then the slots correspond to > cores. If you are in an unmanaged environment, then your hostfile > needs to specify a single hostname, and the slots=x number should > reflect the total number of cores on your machine. > If you then set mpi_paffinity_alone=1, OMPI will bind each rank to its > associated core. > Is that not what you are trying to do? > Ralph I probably didn't explain myself well. In this case, the system is not running a resource manager like SLURM. It is running Linux. If I run numactl --hardware, then I get: available: 8 nodes (0-7) node 0 cpus: 0 1 2 3 node 1 cpus: 4 5 6 7 node 3 cpus: 12 13 14 15 node 4 cpus: 16 17 18 19 node 5 cpus: 20 21 22 23 node 6 cpus: 24 25 26 27 node 7 cpus: 28 29 30 31 where I've elided the memory-related output as well as the node distances. Just to reiterate, each node here is an AMD processor, part of the 8-way system; there is no IP networking going on. What I'd like is, if start a job with mpirun -np 16 , these 16 MPI processes get allocated on continuous "cpus" in numactl parlance, e.g. cpus 0-15, or 12-27, etc. As it stands, if I check the cpus allocated to the aforementioned -np 16 job, I see various cores active on multiple sockets, but I don't see whole sockets (all 4 cores) active at a time on this job. Does this make more sense? -- V. Ram v_r_...@fastmail.fm -- http://www.fastmail.fm - A no graphics, no pop-ups email service
[OMPI users] Processor/core selection/affinity for large shared memory systems
We have an 8-way, 32-core AMD processor machine (single system image) and are at present running OpenMPI 1.2.8 . Jobs are launched locally on the machine itself. As far as I can see, there doesn't seem to be any way to tell OpenMPI to launch the MPI processes on adjacent cores. Presumably such functionality is technically possible via PLPA. Is there in fact a way to specify such a thing with 1.2.8, and if not, will 1.3 support these kinds arguments? Thank you. -- V. Ram v_r_...@fastmail.fm -- http://www.fastmail.fm - Or how I learned to stop worrying and love email again
