Re: [hwloc-users] unusual memory binding results

[Brice Goglin]

Le 25/01/2019 à 14:17, Biddiscombe, John A. a écrit :
> Dear List/Brice
>
> I experimented with disabling the memory touch on threads except for 
> N=1,2,3,4 etc and found a problem in hwloc, which is that the function 
> hwloc_get_area_memlocation was returning '0' when the status of the memory 
> null move operation was -14 (#define EFAULT 14 /* Bad address */). This was 
> when I call get area memlocation immediately after allocating and then 'not' 
> touching. I think if the status is an error, then the function should 
> probably return -1, but anyway. I'll file a bug and send a patch if this is 
> considered to be a bug.


Just to be sure, you talking about move_pages() returning 0 with -14 in
the status array? As opposed to move_pages() returning -1 with errno set
to 14, which would definitely be a bug in hwloc.


When the page is valid but not allocated yet, move_pages() is supposed
to return status = -ENOENT. This case is not an error, so returning 0
with an empty nodeset looks fine to me (pages are not allocated, hence
they are allocated on an empty set of nodes).

-EFAULT means that the page is invalid (you'd get a segfault if you
touch it). I am not sure what we should return in that case. It's also
true that pages are allocated nowhere :)

Anyway, if you get -EFAULT in status, it should mean that an invalid
address was passed to hwloc_get_area_memlocation() or an invalid length.

Brice


___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] unusual memory binding results

[Biddiscombe, John A.]

Dear List/Brice

I experimented with disabling the memory touch on threads except for N=1,2,3,4 
etc and found a problem in hwloc, which is that the function 
hwloc_get_area_memlocation was returning '0' when the status of the memory null 
move operation was -14 (#define EFAULT 14 /* Bad address */). This was when I 
call get area memlocation immediately after allocating and then 'not' touching. 
I think if the status is an error, then the function should probably return -1, 
but anyway. I'll file a bug and send a patch if this is considered to be a bug.

I then modified the test routine to write the value returned from sched_getcpu 
into the touched memory location to verify that the thread binding was doing 
the right thing. The output below from the AMD 8 numanode machine looks good 
with threads 0,8,16 etc each touching memory which follows the pattern expected 
from the 8 numanode test. my get numa domain function however, does not reflect 
the right numanode. It looks correct for the first column (matrices are stored 
in column major order), but after that it falls to pieces. In this test, I'm 
allocating tiles as 512x512 doubles, so 4096 bytes per tile giving one tile 
column per page and I do 512 pages per tile. All the memory locations check out 
and the patters seem fine, but the call to 
// edited version of the one in hwloc source
syscall(__NR_move_pages, 0, 1, pages, nullptr, status, 0) == 0) 
is not returning the numanode that I expect to see from the first touch when it 
is enabled.

Either the syscall is wrong, or the first touch/nexttouch doesn't work (could 
the alloc routine be wrong?)
hwloc_alloc_membind(topo, len, bitmap->get_bmp(),
(hwloc_membind_policy_t)(policy),
flags | HWLOC_MEMBIND_BYNODESET);
where the nodeset should match the numanode mask (I'd will double check that 
right now).

Any ideas on what to try next?

Thanks

JB

get_numa_domain() 8 Domain Numa pattern
00740640
10740640
20740640
30740640
40740640
50740640
60740640
70740640



Contents of memory locations = sched_getcpu()
0 8 16 24 32 40 48 56 
8 16 24 32 40 48 56 0 
16 24 32 40 48 56 0 8 
24 32 40 48 56 0 8 16 
32 40 48 56 0 8 16 24 
40 48 56 0 8 16 24 32 
48 56 0 8 16 24 32 40 
56 0 8 16 24 32 40 48 



Expected 8 Domain Numa pattern
01234567
12345670
23456701
34567012
45670123
56701234
67012345
70123456

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users