Dear Rohith and Naga,

Thank you very much for your quick responses, your information has proven
very useful.

Cheers,
George

On 7 April 2015 at 07:08, Naganarasimha G R (Naga) <
garlanaganarasi...@huawei.com> wrote:

>  Hi George,
>
>  The current implementation present in YARN using Cgroups supports CPU
> isolation but not by pinning to specific cores (Cgroup CPUsets) but based
> on cpu cycles (quota & Period).
> Admin is provided with an option of specifying how much percentage of CPU
> can be used by YARN containers. And Yarn will take care of configuring
> Cgroup Quota and Period files and
> ensures only configured CPU percentage is only used by YARN containers
>
>  Is there any particular need to pin the MR tasks to the specific cores ?
> or you just want to ensure YARN is not using more than the specified
> percentage of CPU in a give node ?
>
>  Regards,
> Naga
>
>  ------------------------------
> *From:* Rohith Sharma K S [rohithsharm...@huawei.com]
> *Sent:* Tuesday, April 07, 2015 09:23
> *To:* user@hadoop.apache.org
> *Subject:* RE: Pin Map/Reduce tasks to specific cores
>
>   Hi George
>
>
>
> In MRV2, YARN supports CGroups implementation.  Using CGroup it is
> possible to run containers in specific cores.
>
>
>
> For your detailed reference, some of the useful links
>
>
> http://dev.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDP2/HDP-2-trunk/bk_system-admin-guide/content/ch_cgroups.html
>
>
> http://blog.cloudera.com/blog/2013/12/managing-multiple-resources-in-hadoop-2-with-yarn/
>
> http://riccomini.name/posts/hadoop/2013-06-14-yarn-with-cgroups/
>
>
>
> P.S : I could not find any related document in Hadoop Yarn docs. I will
> raise ticket for the same  in community.
>
>
>
> Hope the above information will help your use case!!!
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
>
>
> *From:* George Ioannidis [mailto:giorgio...@gmail.com]
> *Sent:* 07 April 2015 01:55
> *To:* user@hadoop.apache.org
> *Subject:* Pin Map/Reduce tasks to specific cores
>
>
>
> Hello. My question, which can be found on *Stack Overflow
> <http://stackoverflow.com/questions/29283213/core-affinity-of-map-tasks-in-hadoop>*
> as well, regards pinning map/reduce tasks to specific cores, either on
> hadoop v.1.2.1 or hadoop v.2.
>
> In specific, I would like to know if the end-user can have any control on
> which core executes a specific map/reduce task.
>
> To pin an application on linux, there's the "taskset" command, but is
> anything similar provided by hadoop? If not, is the Linux Scheduler in
> charge of allocating tasks to specific cores?
>
>
>
> ------------------
>
> Below I am providing two cases to better illustrate my question:
>
> *Case #1:* 2 GiB input size, HDFS block size of 64 MiB and 2 compute
> nodes available, with 32 cores each.
>
> As follows, 32 map tasks will be called; let's suppose that 
> mapred.tasktracker.map.tasks.maximum
> = 16, so 16 map tasks will be allocated to each node.
>
> Can I guarantee that each Map Task will run on a specific core, or is it
> up to the Linux Scheduler?
>
> ------------------
>
> *Case #2:* The same as case #1, but now the input size is 8 GiB, so there
> are not enough slots for all map tasks (128), so multiple tasks will share
> the same cores.
>
> Can I control how much "time" each task will spend on a specific core and
> if it will be reassigned to the same core in the future?
>
> Any information on the above would be highly appreciated.
>
> Kind Regards,
>
> George
>

Reply via email to