On 2/3/17, Brice Goglin <brice.gog...@inria.fr> wrote:
> Le 03/02/2017 21:57, James Elliott a écrit :
>> Brice,
>>
>> Thanks for you comments. I have worked with this some, but this is
>> not working.
>>
>> My goal is to generate images of the cpusets inuse when I run a
>> parallel code using mpirun, aprun, srun, etc... The compute nodes
>> lack the mojo necessary to generate graphical formats, so I can only
>> extract XML on the nodes.
>>
>> I am testing this locally on a 2 NUMA, dual socket workstation with 14
>> cores per socket (so, 28 total cores). I can use OpenMPI to easily
>> spawn/bind processes.
>>
>> E.g.,
>> mpirun --map-by ppr:2:NUMA:pe=7 ./hwloc_plot_mpi.sh
>>
>>
>> hwloc_plot_mpi.sh is very simple:
>>
>> #!/bin/bash
>>
>> pid="$$"
>>
>> rank=${OMPI_COMM_WORLD_RANK}
>>
>> lstopo --pid ${pid} --no-io -f hwloc-${rank}.xml
>>
>> lstopo --pid ${pid} --no-io --append-legend "Rank: ${rank}" -f
>> hwloc-${rank}-orig.png
>>
>> lstopo --append-legend "Rank: ${rank}" --whole-system --input
>> hwloc-${rank}.xml -f hwloc-${rank}.png
>>
>>
>>
>> To test things,
>> 1) write the XML
>> 2) use the same command to write a PNG
>> 3) use the generated XML to generate the PNG
>
> Hello
>
> You're missing --whole-system in 1) and 2)
>
> Also --pid isn't very useful because you're basically looking at the
> current process, and that's the default. The only difference is that the
> process binding is reported in green when using --pid. Does it matter?
> See below.
>
>>
>> (2) and (3) should produce the same image if I am doing things correctly.
>>
>> The image for (2) is unique for each process, showing 7 *different*
>> cores shaded in each figure (4 images are generated since I spawn 4
>> processes)
>> The images from (3) are all identical (no shading)
>
> What do you mean with shaded? Red or green?
>
> Red means unavailable. Requires --whole-system everywhere.
>
> Green means that's where the process is bound. But XML doesn't store the
> information about where the process is bound, so you may only get Green
> in 2).
>
This is exactly what I am attempting to do (and finding it does not work).
I would like to have a figure with green shadings so that I have a
visual representation of where my MPI process lived on the machine.
I currently have a function (in C) that I use in my codes that
inspects affinities, but when I discuss app performance with others, I
would like to be able to show (graphically) exactly how their app uses
the resources. I work mostly with hybrid MPI/OpenMP codes, developed
by smart scientists who are not familiar with things like affinity.
>>
>> To test without MPI, you would just need to set a processes affinity
>> and then use its PID instead.
>>
>> What I see, is that the XML generated in (1) is identical for all MPI
>> processes, even though they have different PIDs and different CPUSETS.
>
> Are you talking about different MPI runs, or different MPI ranks within
> the same run?
>
> My feeling is that you think you should be seeing different cpusets for
> each process, but they actually have the same cpuset but different
> bindings. Cores outside the cpuset are red when --whole-system, or
> totally ignored otherwise.
>
> In (2), you don't have --whole-system, no red cores. But you have --pid,
> so you get one green core per process, it's its binding. That's why you
> get different images for each process.
> in (3), you inherit the missing --whole-system from (1) through XML, no
> red cores either. But XML doesn't save the process binding, no green
> cores either. Same image for each process.
>
>
> Do you care about process binding (what mpirun applies to each rank?) or
> about cpusets (what the batch scheduler applies to the entire job before
> mpirun?)
>
> If cpuset, just add --whole-system everywhere, it should be enough.
> If binding, there's no direct way with lstopo (but we have a way to save
> custom colors for individual objects in the XML).
>
> Brice
>
>
_______________________________________________
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
