Dear list, I'm allocating a matrix of size (say) 2048*2048 on a node with 2 numa domains and initializing the matrix by using 2 threads, one pinned on each numa domain - with the idea that I can create tiles of memory bound to each numa domain rather than having pages assigned all to one, interleaved, or possibly random. The tiling pattern can be user defined, but I am using a simple strategy that touches pages based on a simple indexing scheme using (say) a tile size of 256 elements and should give a pattern like this
Expected 2 Domain Numa pattern 0000111100001111 0000111100001111 0000111100001111 0000111100001111 1111000011110000 1111000011110000 1111000011110000 1111000011110000 0000111100001111 0000111100001111 0000111100001111 0000111100001111 1111000011110000 1111000011110000 1111000011110000 1111000011110000 Where the 0's and 1's correspond to the numa node that touches the block of memory. The memory is allocated using HWLOC_MEMBIND_FIRSTTOUCH (I also tried HWLOC_MEMBIND_NEXTTOUCH) and calls hwloc_alloc_membind_nodeset( ... ); On broadwell nodes (linux kernel 4.4.103-6.38_4.0.153-cray_ari_c), it seems to mostly work and when I display the memory binding using a call to hwloc_get_area_memlocation( ... ) I see a pattern that matches the one above. However, I do occasionally see 1's and 0's that are incorrect. When I run the same code on a login node, Haswell (4.4.156-94.61.1.16335.0.PTF.1107299-default), I generally see patterns that are more like 0010101010101010 0010101010101010 0010101010101010 0010101010101010 0010101010101010 0010101010101010 0010101010101010 0010101010101010 0010101010101010 0010101010101010 0010101010101010 0010101010101010 0010101010101010 0010101010101010 0010101010101010 0010101010101010 and are clearly wrong. Testing on an AMD AMD EPYC 7501 32-Core, node (running 3.10.0-957.1.3.el7.x86_64), I should see a pattern of 8 nodes such as Expected 8 Domain Numa pattern 0011223344556677 0011223344556677 1122334455667700 1122334455667700 2233445566770011 2233445566770011 3344556677001122 3344556677001122 4455667700112233 4455667700112233 5566770011223344 5566770011223344 6677001122334455 6677001122334455 7700112233445566 7700112233445566 but I'm actually seeing 0021322302001122 0021322302001122 1021322302001122 1021322302001122 2021322302001122 2021322302001122 3021322302001122 3021322302001122 4021322302001122 4021322302001122 5021322302001122 5021322302001122 6021322302001122 6021322302001122 7021322302001122 7021322302001122 I've checked and triple checked the thread bindings and address mappings and am 99% certain that the fault is either in the get_area_memlocation or in the touch pages not actually causing the page to be bound as expected. Can anyone suggest what might be my problem - or a test I might try to help narrow down what's wrong. Many thanks JB -- Dr. John Biddiscombe, email:biddisco @.at.@ cscs.ch http://www.cscs.ch/ CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07 Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82 _______________________________________________ hwloc-users mailing list hwloc-users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/hwloc-users
