Ralph and Brice,
since
https://github.com/open-mpi/ompi/commit/fe68f2309912ea2afdc3339ff9a3b697f69a2dd1
we likely set the default binding policy to OPAL_BIND_TO_NUMA
unfortunatly, that does not work on my VM (Virtual Box, single socket, 4
cores) since there is no HWLOC_OBJ_NODE here
with hwloc v1.11
<topology>
<object type="Machine" os_index="0" cpuset="0x0000000f"
complete_cpuset="0x0000000f" online_cpuset="0x0000000f"
allowed_cpuset="0x0000000f" local_memory="3975217152">
<object type="Package" os_index="0" cpuset="0x0000000f"
complete_cpuset="0x0000000f" online_cpuset="0x0000000f"
allowed_cpuset="0x0000000f">
<object type="Cache" cpuset="0x00000001"
complete_cpuset="0x00000001" online_cpuset="0x00000001"
allowed_cpuset="0x00000001" cache_size="8388608" depth="3"
cache_linesize="64" cache_associativity="16" cache_type="0">
<object type="Cache" cpuset="0x00000001"
complete_cpuset="0x00000001" online_cpuset="0x00000001"
allowed_cpuset="0x00000001" cache_size="262144" depth="2"
cache_linesize="64" cache_associativity="8" cache_type="0">
<object type="Cache" cpuset="0x00000001"
complete_cpuset="0x00000001" online_cpuset="0x00000001"
allowed_cpuset="0x00000001" cache_size="32768" depth="1"
cache_linesize="64" cache_associativity="8" cache_type="1">
<object type="Cache" cpuset="0x00000001"
complete_cpuset="0x00000001" online_cpuset="0x00000001"
allowed_cpuset="0x00000001" cache_size="32768" depth="1"
cache_linesize="64" cache_associativity="8" cache_type="2">
<object type="Core" os_index="0" cpuset="0x00000001"
complete_cpuset="0x00000001" online_cpuset="0x00000001"
allowed_cpuset="0x00000001">
<object type="PU" os_index="0" cpuset="0x00000001"
complete_cpuset="0x00000001" online_cpuset="0x00000001"
allowed_cpuset="0x00000001"/>
[...]
</topology>
but with the latest hwloc (master branch, that does not work yet with
Open MPI)
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE topology SYSTEM "hwloc.dtd">
<topology>
<object type="Machine" os_index="0" cpuset="0x0000000f"
complete_cpuset="0x0000000f" allowed_cpuset="0x0000000f"
nodeset="0x00000001" complete_nodeset="0x00000001"
allowed_nodeset="0x00000001" gp_index="1">
<object type="NUMANode" os_index="0" cpuset="0x0000000f"
complete_cpuset="0x0000000f" allowed_cpuset="0x0000000f"
nodeset="0x00000001" complete_nodeset="0x00000001"
allowed_nodeset="0x00000001" gp_index="2" local_memory="4294500352">
<object type="Package" os_index="0" cpuset="0x0000000f"
complete_cpuset="0x0000000f" allowed_cpuset="0x0000000f"
nodeset="0x00000001" complete_nodeset="0x00000001"
allowed_nodeset="0x00000001" gp_index="3">
<object type="L3Cache" cpuset="0x00000001"
complete_cpuset="0x00000001" allowed_cpuset="0x00000001"
nodeset="0x00000001" complete_nodeset="0x00000001"
allowed_nodeset="0x00000001" gp_index="9" cache_size="8388608" depth="3"
cache_linesize="64" cache_associativity="16" cache_type="0">
<object type="L2Cache" cpuset="0x00000001"
complete_cpuset="0x00000001" allowed_cpuset="0x00000001"
nodeset="0x00000001" complete_nodeset="0x00000001"
allowed_nodeset="0x00000001" gp_index="8" cache_size="262144" depth="2"
cache_linesize="64" cache_associativity="8" cache_type="0">
<object type="L1Cache" cpuset="0x00000001"
complete_cpuset="0x00000001" allowed_cpuset="0x00000001"
nodeset="0x00000001" complete_nodeset="0x00000001"
allowed_nodeset="0x00000001" gp_index="6" cache_size="32768" depth="1"
cache_linesize="64" cache_associativity="8" cache_type="1">
<object type="L1iCache" cpuset="0x00000001"
complete_cpuset="0x00000001" allowed_cpuset="0x00000001"
nodeset="0x00000001" complete_nodeset="0x00000001"
allowed_nodeset="0x00000001" gp_index="7" cache_size="32768" depth="1"
cache_linesize="64" cache_associativity="8" cache_type="2">
<object type="Core" os_index="0" cpuset="0x00000001"
complete_cpuset="0x00000001" allowed_cpuset="0x00000001"
nodeset="0x00000001" complete_nodeset="0x00000001"
allowed_nodeset="0x00000001" gp_index="4">
<object type="PU" os_index="0" cpuset="0x00000001"
complete_cpuset="0x00000001" allowed_cpuset="0x00000001"
nodeset="0x00000001" complete_nodeset="0x00000001"
allowed_nodeset="0x00000001" gp_index="5"/>
[...]
</topology>
an ugly work around is to bind to HWLOC_OBJ_PACKAGE when there is no
HWLOC_OBJ_NODE
diff --git a/orte/mca/rmaps/base/rmaps_base_binding.c
b/orte/mca/rmaps/base/rmaps_base_binding.c
index 6786da7..ddc106c 100644
--- a/orte/mca/rmaps/base/rmaps_base_binding.c
+++ b/orte/mca/rmaps/base/rmaps_base_binding.c
@@ -916,6 +916,10 @@ int orte_rmaps_base_compute_bindings(orte_job_t *jdata)
bind_depth =
hwloc_get_cache_type_depth(node->topology, clvl,
(hwloc_obj_cache_type_t)-1);
} else {
bind_depth = hwloc_get_type_depth(node->topology, hwb);
+ if (0 > bind_depth && HWLOC_OBJ_NODE == hwb) {
+ hwb = HWLOC_OBJ_PACKAGE;
+ bind_depth = hwloc_get_type_depth(node->topology, hwb);
+ }
}
if (0 > bind_depth) {
/* didn't find such an object */
or we could check the existence of HWLOC_OBJ_PACKAGE before setting the
default policy in orte_rmaps_base_map_job(),
not to mention it is possible to generate an xml topology with the
latest hwloc, and
mpirun --mca hwloc_base_topo_file topo.xml ...
Brice,
things would be much easier if there were an HWLOC_OBJ_NODE object in
the topology.
could you please consider backporting the relevant changes from master
into the v1.11 branch ?
Cheers,
Gilles
_______________________________________________
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
