Hello,
What do you mean with " it can not bind the specified memory section
(addr, len) to the desired NUMA node"?
Did it fail? If so, what does errno contain?
If it didn't fail, what did it do instead?
thanks
Brice
Le 24/04/2016 23:02, Rezaul Karim Raju a écrit :
> Hi ...
>
> I was trying to bind each quarter of an array to 4 different NUMA
> nodes, and doing as below:
>
> *//ALLOCATION *
> *obj_a = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 0);*
>
> *array =* hwloc_alloc_membind_nodeset( topology, size, obj_a->nodeset,
> HWLOC_MEMBIND_BIND, 1);
> *tmp *= hwloc_alloc_membind_nodeset( topology, size, obj_a->nodeset,
> HWLOC_MEMBIND_BIND, 1);
> *
> *
> *// DISTRIBUTED BINDING [my system has 8 NUMA nodes (0-7)]*
> printf("Address of array:= %p, array+size/4:= %p, array+size/2:= %p,
> array+3*size/4:= %p \n", array, array+size/4, array+size/2,
> array+3*size/4);
> // bind 1st quarter to node (n-1)
> hwloc_set_area_membind_nodeset(topology, (array), size/4,
> obj_a->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
> hwloc_set_area_membind_nodeset(topology, (tmp), size/4,
> obj_a->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
> // bind 2nd quarter to node (2)
> *obj_b = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 2);*
> hwloc_set_area_membind_nodeset(topology, (array+size/4), size/4,
> obj_b->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
> hwloc_set_area_membind_nodeset(topology, (tmp +size/4), size/4,
> obj_b->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
>
> // bind 3rd quarter to node (4)
> *obj_c = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 4);*
> hwloc_set_area_membind_nodeset(topology, array+size/2, size/4,
> obj_c->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
> hwloc_set_area_membind_nodeset(topology, tmp+size/2, size/4,
> obj_c->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
> // bind 4th quarter to node (6)
> *obj_d = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 6);*
> hwloc_set_area_membind_nodeset(topology, array+3*size/4, size/4,
> obj_d->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
> hwloc_set_area_membind_nodeset(topology, tmp+3*size/4, size/4,
> obj_d->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
>
>
> My intention here is to distribute 'array' (which is - long type
> element:
> array = (ELM *) malloc(bots_arg_size * sizeof(ELM));
> tmp = (ELM *) malloc(bots_arg_size * sizeof(ELM));) over nodes through
> hwloc memory binding.
>
> 1). But except only *obj_a, it can not bind the specified memory
> section (addr, len) to the desired NUMA node. *
> 2). I did tried with MEMBIND_INTERLEAVE policy
> array = hwloc_alloc_membind_nodeset(topology, size, cset_available,
> HWLOC_MEMBIND_INTERLEAVE, HWLOC_MEMBIND_MIGRATE);
> tmp = hwloc_alloc_membind_nodeset(topology, size, cset_available,
> HWLOC_MEMBIND_INTERLEAVE, HWLOC_MEMBIND_MIGRATE);
> but I did get it working here as well.
>
>
> *Can you please comment on this..? *
>
> Thank you very much in advance..!!
> - Raju
>
> On Mon, Mar 21, 2016 at 11:25 AM, Rezaul Karim Raju
> <raju.cse.b...@gmail.com <mailto:raju.cse.b...@gmail.com>> wrote:
>
> Thanks, Brice.!
>
> On Mon, Mar 21, 2016 at 11:22 AM, Brice Goglin
> <brice.gog...@inria.fr <mailto:brice.gog...@inria.fr>> wrote:
>
> For testing, you can use this tarball:
>
> https://ci.inria.fr/hwloc/job/zcustombranch-0-tarball/lastSuccessfulBuild/artifact/hwloc-getmemlocation-20160320.2208.gitd2f6537.tar.gz
>
>
>
>
>
> Le 21/03/2016 17:21, Rezaul Karim Raju a écrit :
>> Hi Brice,
>>
>> Thanks for your email.
>> I believe it is definitely helpful. Getting memory range
>> within the current process will be very good information to
>> drill down.
>> Let me use this and I will get back if any
>> clarification/comment I have.
>>
>> Regards-
>> Raju
>>
>> On Sun, Mar 20, 2016 at 4:26 PM, Brice Goglin
>> <brice.gog...@inria.fr <mailto:brice.gog...@inria.fr>> wrote:
>>
>> I just pushed a proposal, see
>> https://github.com/open-mpi/hwloc/issues/97
>>
>> Brice
>>
>>
>>
>>
>> Le 18/12/2015 20:45, Brice Goglin a écrit :
>>> Yes, we're "thinking" about it. But there are open
>>> questions as mentioned in the github issue.
>>> By the way, we wouldn't return NULL in case of
>>> non-physically-allocated buffer, but rather set the
>>> output nodeset to 0.
>>> You should comment on the issue directly, instead of
>>> replying here, otherwise your comments may get lost.
>>>
>>> Brice
>>>
>>>
>>>
>>>
>>> Le 18/12/2015 18:57, Rezaul Karim Raju a écrit :
>>>> Hi Brice,
>>>>
>>>> Thanks for your time and nice explanation.
>>>> I have looked at the issue with location return (the
>>>> page proportion across multiple node & physical
>>>> allocation). Are you thinking to add this function..?
>>>> Like if we think list of node or nodes where the array
>>>> is allocated (only if physically allocated otherwise
>>>> NULL) is it possible..?
>>>>
>>>> I am looking for getting the physical location of data
>>>> allocated by OS default policy. Appreciate any better
>>>> idea and please share with me.
>>>>
>>>> Best Regards,
>>>> - Raju
>>>>
>>>> On Tue, Dec 15, 2015 at 3:28 AM, Brice Goglin
>>>> <brice.gog...@inria.fr <mailto:brice.gog...@inria.fr>>
>>>> wrote:
>>>>
>>>>
>>>>
>>>> Le 15/12/2015 07:21, Brice Goglin a écrit :
>>>>>
>>>>>
>>>>> Le 15/12/2015 05:57, Rezaul Karim Raju a écrit :
>>>>>> *OUTPUT: *
>>>>>> *Policy-->* buffer(Array: A) *membind [default OS
>>>>>> binding] Policy is:= 1 [1 refers
>>>>>> to *HWLOC_MEMBIND_FIRSTTOUCH
>>>>>>
>>>>>> <https://www.open-mpi.org/projects/hwloc/doc/v1.11.1/a00083.php#ggac9764f79505775d06407b40f5e4661e8a979c7aa78dd32780858f30f47a72cca0>*]*
>>>>>> *Nodeset --> *buffer(Array: A) bound to
>>>>>> nodeset*0x000000ff *with contains*:*
>>>>>> * *node #0 (OS index 0) with 8387047424 bytes of
>>>>>> memory
>>>>>> node #1 (OS index 1) with 8471617536 bytes of memory
>>>>>> node #2 (OS index 2) with 8471621632 bytes of memory
>>>>>> node #3 (OS index 3) with 8471617536 bytes of memory
>>>>>> node #4 (OS index 4) with 8471621632 bytes of memory
>>>>>> node #5 (OS index 5) with 8471617536 bytes of memory
>>>>>> node #6 (OS index 6) with 8471621632 bytes of memory
>>>>>> node #7 (OS index 7) with 8471564288 bytes of memory
>>>>>> *
>>>>>> *
>>>>>> *why it shows allocated memory is bound to all
>>>>>> available nodeset..? should it not be allocated
>>>>>> to a specific nodeset one ..?*
>>>>>> *
>>>>>> *
>>>>>
>>>>> Hello
>>>>>
>>>>> You are confusing the "binding" and the "actual
>>>>> location". Your memory buffer isn't bound to a
>>>>> specific location by default. But Linux has to
>>>>> allocate it somewhere. So your buffer is "located"
>>>>> in some node after the allocation, but it is not
>>>>> "bound" there (what get_area_membind returns)
>>>>> which means Linux could have allocated it
>>>>> somewhere else.
>>>>>
>>>>> hwloc cannot currently return the "location" of a
>>>>> memory buffer. I have been thinking about adding
>>>>> this feature in the past, but it looks like you
>>>>> are the first actual user requesting this. We
>>>>> could add something like
>>>>> hwloc_get_last_memory_location(topology, input
>>>>> buffer, outputnodeset)
>>>>> At least on Linux that's possible.
>>>>>
>>>>> For clarity, this is similar to the difference
>>>>> between hwloc_get_cpubind() and
>>>>> hwloc_get_last_cpu_location(): A task always runs
>>>>> on a specific PU, even if it is not bound to
>>>>> anything specific PU.
>>>>
>>>> By the way, there is already an issue for this:
>>>> https://github.com/open-mpi/hwloc/issues/97
>>>>
>>>> Feel to comment there.
>>>>
>>>> Brice
>>>>
>>>>
>>>> _______________________________________________
>>>> hwloc-users mailing list
>>>> hwloc-us...@open-mpi.org
>>>> <mailto:hwloc-us...@open-mpi.org>
>>>> Subscription:
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
>>>> Link to this post:
>>>>
>>>> http://www.open-mpi.org/community/lists/hwloc-users/2015/12/1226.php
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> ------------------------
>>>> RaJu, Rezaul Karim
>>>> Graduate Student (PhD) | Computer Science | University
>>>> of Houston
>>>> Research in High Performance Computing Tools
>>>> Houston, Texas-77004
>>>
>>
>>
>>
>>
>> --
>> ------------------------
>> RaJu, Rezaul Karim
>> Graduate Student (PhD) | Computer Science | University of
>> Houston
>> Research in High Performance Computing Tools
>> Houston, Texas-77004
>
>
>
>
> --
> ------------------------
> RaJu, Rezaul Karim
> Graduate Student (PhD) | Computer Science | University of Houston
> Research in High Performance Computing Tools
> Houston, Texas-77004
>
>
>
>
> --
> ------------------------
> RaJu, Rezaul Karim
> Graduate Student (PhD) | Computer Science | University of Houston
> Research in High Performance Computing Tools
> Houston, Texas-77004
