[
https://issues.apache.org/jira/browse/YARN-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106995#comment-14106995
]
Varun Vasudev commented on YARN-160:
------------------------------------
[~djp]
{quote}
Both physical id and core id are not guaranteed to have in /proc/cpuinfo
(please see below for my local VM's info). We may use processor number instead
in case these ids are 0 (like we did in Windows). Again, this weak my
confidence that this automatic way of getting CPU/memory resources should
happen by default (not sure if any cross-platform issues). May be a safer way
here is to keep previous default behavior (with some static setting) with an
extra config to enable this. We can wait this feature to be more stable later
to change the default behavior.
{noformat}
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 70
model name : Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
stepping : 1
cpu MHz : 2295.265
cache size : 6144 KB
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm
constant_tsc up arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc
aperfmperf unfair_spinlock pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2
x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm ida arat epb
xsaveopt pln pts dts tpr_shadow vnmi ept vpid fsgsbase smep
bogomips : 4590.53
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
{noformat}
{quote}
In the example you gave, where we have processors listed but no physical id or
core id entries, the numProcessors will be set to the number of entries and
numCores will be set to 1. From the diff -
{noformat}
+ numCores = 1;
{noformat}
There is also a test case to ensure this behaviour.
In addition, cluster administrators can decide whether the NodeManager should
report numProcessors or numCores by toggling
yarn.nodemanager.resource.count-logical-processors-as-vcores which by default
is true. In the vm example, by default the NodeManager will report vcores as
the number of processor entries in /proc/cpuinfo. If
yarn.nodemanager.resource.count-logical-processors-as-vcores is set to false,
the NodeManager will report vcores as 1(if there are no physical id or core id
entries).
> nodemanagers should obtain cpu/memory values from underlying OS
> ---------------------------------------------------------------
>
> Key: YARN-160
> URL: https://issues.apache.org/jira/browse/YARN-160
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.0.3-alpha
> Reporter: Alejandro Abdelnur
> Assignee: Varun Vasudev
> Fix For: 2.6.0
>
> Attachments: apache-yarn-160.0.patch, apache-yarn-160.1.patch
>
>
> As mentioned in YARN-2
> *NM memory and CPU configs*
> Currently these values are coming from the config of the NM, we should be
> able to obtain those values from the OS (ie, in the case of Linux from
> /proc/meminfo & /proc/cpuinfo). As this is highly OS dependent we should have
> an interface that obtains this information. In addition implementations of
> this interface should be able to specify a mem/cpu offset (amount of mem/cpu
> not to be avail as YARN resource), this would allow to reserve mem/cpu for
> the OS and other services outside of YARN containers.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
