Hi Brice, thanks for the quick reply! I was able to get access to one KNL server and I have tested it there. It works - bandwidth is annotated and reported:-) See [1]
It's also possible to add memory attribute using the C API or with > hwloc-annotate to modify a XML This is interesting! ACPI tables are often wrong - having the option to annotate more accurate data to the hwloc is great. We have a simple C program to measure the bandwidth between NUMA nodes, producing a table similar to the output of numactl -H (but with values in GB/s). node 0 1 2 3 0: 10 16 16 16 1: 16 10 16 16 2: 16 16 10 16 3: 16 16 16 10 I was trying to annotate it using hwloc-annotate, but I have not succeeded. : lstopo in.xml hwloc-annotate in.xml out.xml node:0 memattr bandwidth node:0 18 Failed to find memattr by name bandwidth Is there some example of how to do this? Also, are there any plans for having a tool, which would measure the memory bandwidth and annotate the results to XML for later usage with hwloc commands? There are 4 standard attributes defined in hwloc/memattrs.h: capacity, > locality, latency and bandwidth.They are also visible in lstopo -vv or > lstopo --memattrs. I'll something in the doc. Thanks for the hint! Adding pointer to "lstopo --memattrs" to "hwloc-info -h" output would be IMHO sufficient: --best-memattr <attr> Only display the best target among the local nodes. See the output of "lstopo --memattrs" for the list of supported attributes. Thanks a lot! Jirka [root@intel-knightslanding-01 hwloc-2.3.0]# utils/lstopo/lstopo-no-graphics --memattrs Memory attribute #0 name `Capacity' flags 1 NUMANode L#0 = 33417318400 NUMANode L#1 = 16910688256 Memory attribute #1 name `Locality' flags 2 NUMANode L#0 = 272 NUMANode L#1 = 272 Memory attribute #2 name `Bandwidth' flags 5 NUMANode L#0 = 90000 from cpuset 0x0000ffff,0xffffffff,0xffffffff,0xffffffff,0xffffffff,0xffffffff,0xffffffff,0xffffffff,0xffffffff (Machine L#0) NUMANode L#1 = 360000 from cpuset 0x0000ffff,0xffffffff,0xffffffff,0xffffffff,0xffffffff,0xffffffff,0xffffffff,0xffffffff,0xffffffff (Machine L#0) On Thu, Oct 1, 2020 at 7:28 PM Brice Goglin <brice.gog...@inria.fr> wrote: > > Le 01/10/2020 à 19:16, Jirka Hladky a écrit : > > Hi Brice, > > this new feature sounds very interesting! > > Add hwloc/memattrs.h for exposing latency/bandwidth information >> between initiators (CPU sets for now) and target NUMA nodes, >> typically on heterogeneous platforms. > > > If I get it right, I need to have an ACPI HMAT table on the system to use > the new functionality, right? > > > Hello Jirka > > It's also possible to add memory attribute using the C API or with > hwloc-annotate to modify a XML (you may create attribute, or add values for > a given attribute). > > > I have tried following on Fedora > acpidump -o acpidump.bin > acpixtract -a acpidump.bin > > but there is no HMAT table reported. So it seems I'm out of luck, and I > cannot test the new functionality, right? > > > Besides KNL (which is too old to have HMAT, but hwloc now provides > hardwired bandwidth/latency values), the only platforms with heterogeneous > memories right now are Intel machines with Optane DCPMM (NVDIMMs). Some > have a HMAT, some don't. If your machine doesn't, it's possible to provide > a custom HMAT table in the initrd. That's not easy, so adding attribute > values with hwloc-annotate might be easier. > > > > Also, where can we find the list of attributes supported by --best-memattr? > --best-memattr <attr> Only display the best target among the local nodes > > > There are 4 standard attributes defined in hwloc/memattrs.h: capacity, > locality, latency and bandwidth.They are also visible in lstopo -vv or > lstopo --memattrs. I'll something in the doc. > > > > By trial and error, I have found out that latency and bandwidth are > supported. Are there any other? Could you please add the list to hwloc-info > -h? > > > I could add the default ones, but I'll need to specify that additional > user-given attributes may exist. > > Thanks for the feedback. > > Brice > > > > > hwloc-info --best-memattr bandwidth > hwloc-info --best-memattr latency > > Thanks a lot! > Jirka > > > On Thu, Oct 1, 2020 at 12:45 AM Brice Goglin <brice.gog...@inria.fr> > wrote: > >> hwloc (Hardware Locality) 2.3.0 is now available for download. >> >> https://www.open-mpi.org/software/hwloc/v2.3/ >> <https://www.open-mpi.org/software/hwloc/v2.0/> >> >> v2.3.0 brings quite a lot of changes. The biggest one is the addition >> of the memory attribute API to expose hardware information that vendors >> are (slowly) adding to ACPI tables to describe heterogeneous memory >> platforms (mostly DDR+NVDIMMs right now). >> >> The following is a summary of the changes since v2.2.0. >> >> Version 2.3.0 >> ------------- >> * API >> + Add hwloc/memattrs.h for exposing latency/bandwidth information >> between initiators (CPU sets for now) and target NUMA nodes, >> typically on heterogeneous platforms. >> - When available, bandwidths and latencies are read from the ACPI HMAT >> table exposed by Linux kernel 5.2+. >> - Attributes may also be customized to expose user-defined performance >> information. >> + Add hwloc_get_local_numanode_objs() for listing NUMA nodes that are >> local to some locality. >> + The new topology flag HWLOC_TOPOLOGY_FLAG_IMPORT_SUPPORT causes >> support arrays to be loaded from XML exported with hwloc 2.3+. >> - hwloc_topology_get_support() now returns an additional "misc" >> array with feature "imported_support" set when support was imported. >> + Add hwloc_topology_refresh() to refresh internal caches after modifying >> the topology and before consulting the topology in a multithread context. >> * Backends >> + Add a ROCm SMI backend and a hwloc/rsmi.h helper file for getting >> the locality of AMD GPUs, now exposed as "rsmi" OS devices. >> Thanks to Mike Li. >> + Remove POWER device-tree-based topology on Linux, >> (it was disabled by default since 2.1). >> * Tools >> + Command-line options for specifying flags now understand comma-separated >> lists of flag names (substrings). >> + hwloc-info and hwloc-calc have new --local-memory --local-memory-flags >> and --best-memattr options for reporting local memory nodes and filtering >> by memory attributes. >> + hwloc-bind has a new --best-memattr option for filtering by memory >> attributes >> among the memory binding set. >> + Tools that have a --restrict option may now receive a nodeset or >> some custom flags for restricting the topology. >> + lstopo now has a --thickness option for changing line thickness in the >> graphical output. >> + Fix lstopo drawing when autoresizing on Windows 10. >> + Pressing the F5 key in lstopo X11 and Windows graphical/interactive >> outputs >> now refreshes the display according to the current topology and binding. >> + Add a tikz lstopo graphical backend to generate picture easily included >> into >> LaTeX documents. Thanks to Clement Foyer. >> * Misc >> + The default installation path of the Bash completion file has changed to >> ${datadir}/bash-completion/completions/hwloc. Thanks to Tomasz Kłoczko. >> >> >> Changes since 2.3.0rc1 are negligible. >> -- >> Brice >> >> >> _______________________________________________ >> hwloc-announce mailing list >> hwloc-annou...@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/hwloc-announce > > > > -- > -Jirka > > _______________________________________________ > hwloc-users mailing > listhwloc-us...@lists.open-mpi.orghttps://lists.open-mpi.org/mailman/listinfo/hwloc-users > > _______________________________________________ > hwloc-users mailing list > hwloc-users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/hwloc-users -- -Jirka
_______________________________________________ hwloc-users mailing list hwloc-users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/hwloc-users
