Re: [hwloc-devel] hwloc_distances as utility?
Le 03/07/2011 23:55, Jiri Hladky a écrit : > Hi all, > > I have come across tests/hwloc_distances test and I believe that it > would be great to convert this into the utility > "hwloc-report-instances" published under utils/ directory. Please let > me know what you think about it. > > It would take the same input as hwloc-info (read topology from > different formats instead of discovering the topology on the local > machine), support both logical and physical indexes (-l and -p) switch. Hello Jirka, I added a hwloc-distances utility to the trunk today. It will be in the next nightly snapshot (if doxygen problems are fixed on the build machine). Brice
Re: [hwloc-devel] hwloc_distances as utility?
Le 05/07/2011 22:04, Jiri Hladky a écrit : > Well, this is interesting. numactl --hardware shows the number of > hops, regarding to the information from that private BZ. I think this is wrong. numactl takes everything from sysfs as far as I can tell. On x86, sysfs distances are ACPI SLIT latencies (memory latencies that are normalized to 10 for latencies from one cpu to its local memory). A couple months ago, I checked all Linux ports that show distances in sysfs. All of them report memory latencies, except the SGI IP27 as mentioned previously (this one indeed shows number of hops (0 when local) and it makes a lot of sense for this architecture). That said, the latency that is reported in sysfs is definitely related to the number of hops, and I assume that BIOS vendors look at the number of hops when deciding what to put in the ACPI SLIT distances. It's not something measured at boot. It's kind of hardcoded by/in the BIOS and decided as the same time the routing is computed. But still the ACPI spec says that distances are normalized latencies. One problem I see with the number of hops is that it doesn't make sense on some machines. On some 8-socket AMD machines (such as 8amd64-4n2c.tar.bz2 below) , the hypertransport route between some sockets varies with the type of packet (response or request) and the direction. So the number of hops ends up being asymmetric, depends on read/write, and can be half of an integer. > could me send me the output of hwloc-gather-topology for some NUMA > box? I don't have access to any NUMA running a recent version of kernel. Look at tests/linux/ in the hwloc SVN. The following tarballs contain NUMA architectures. Some of these were gathered while running old kernels, but I don't think it matters because Linux/sysfs reports what the BIOS without changing much of it. 8amd64-4n2c.tar.bz2 (2 hops, but distance is reported as local(10) or remote (20)) 16amd64-8n2c.tar.bz2 (up to 3 hops, but local or remote too) 48amd64-4d2n6c-sparse.tar.bz2 (2 hops IIRC, local(10), remote(16) or "very remote"(22)) 96em64t-4n4d3ca2co.tar.bz2 (1 hop, local(10) or remote(22)) 16ia64-8n2s.tar.bz2 (2 hops, 10/25/29) 256ia64-64n2s2c.tar.bz2 (big SGI Altix machine with several hops, 10/22/26/30/34) Brice
Re: [hwloc-devel] hwloc_distances as utility?
Hi Brice, > By the way, lstopo shows distance information, but it does not change it > depending on -l/-p. We may want to fix this. > Thanks for the hint, I din't know it. > Actually, Linux only uses the number of hops on one specific MIPS > architecture (SGI IP27 Origin 200/2000). In other cases, it uses the > cpu-to-memory latency (usually reported by ACPI or so). > Well, this is interesting. numactl --hardware shows the number of hops, regarding to the information from that private BZ. Where does hwloc takes the distance information? Is it stored with hwloc-gather-topology script? If yes, could me send me the output of hwloc-gather-topology for some NUMA box? I don't have access to any NUMA running a recent version of kernel. > > > On some systems number of hops does not represent memory bandwidth. I > > have reported this in BZ 655041 > > > > https://bugzilla.redhat.com/show_bug.cgi?id=655041 > > This bug is private unfortunately. > > > In any case I believe that hwloc-report-instances would be useful > > utility. Please let me know your opinion. > > Agreed. > OK, I will try to implement it as time permits. Thanks Jirka
Re: [hwloc-devel] hwloc_distances as utility?
Le 03/07/2011 23:55, Jiri Hladky a écrit : > Hi all, > > I have come across tests/hwloc_distances test and I believe that it > would be great to convert this into the utility > "hwloc-report-instances" published under utils/ directory. Please let > me know what you think about it. > > It would take the same input as hwloc-info (read topology from > different formats instead of discovering the topology on the local > machine), support both logical and physical indexes (-l and -p) switch. By the way, lstopo shows distance information, but it does not change it depending on -l/-p. We may want to fix this. > I have used stream memory bandwidth benchmark > (http://www.cs.virginia.edu/stream/) in the past to produce the > similar output as tests/hwloc_distances. It was interesting to see > that numactl and kernel scheduler are both using number of hopes > instead of memory bandwidth. Actually, Linux only uses the number of hops on one specific MIPS architecture (SGI IP27 Origin 200/2000). In other cases, it uses the cpu-to-memory latency (usually reported by ACPI or so). > On some systems number of hops does not represent memory bandwidth. I > have reported this in BZ 655041 > > https://bugzilla.redhat.com/show_bug.cgi?id=655041 This bug is private unfortunately. > In any case I believe that hwloc-report-instances would be useful > utility. Please let me know your opinion. Agreed. There are still several things to improve regarding distances. Everything should be in https://svn.open-mpi.org/trac/hwloc/ticket/43 Brice
[hwloc-devel] hwloc_distances as utility?
Hi all, I have come across tests/hwloc_distances test and I believe that it would be great to convert this into the utility "hwloc-report-instances" published under utils/ directory. Please let me know what you think about it. It would take the same input as hwloc-info (read topology from different formats instead of discovering the topology on the local machine), support both logical and physical indexes (-l and -p) switch. I have used stream memory bandwidth benchmark ( http://www.cs.virginia.edu/stream/) in the past to produce the similar output as tests/hwloc_distances. It was interesting to see that numactl and kernel scheduler are both using number of hopes instead of memory bandwidth. On some systems number of hops does not represent memory bandwidth. I have reported this in BZ 655041 https://bugzilla.redhat.com/show_bug.cgi?id=655041 In any case I believe that hwloc-report-instances would be useful utility. Please let me know your opinion. Thanks Jirka
