Hi Ravi,
sorry for late answer. :) We are on hadoop 2.6-cdh5.7.
Cheers,
Jan
On 12.8.2016 01:57, Ravi Prakash wrote:
Hi Jan!
Yes! Makes sense. I'm sure there were bigger changes for the
ResourceHandler. Which version are you on?
Cheers
Ravi
On Thu, Aug 11, 2016 at 7:48 AM, Jan Lukavský
<jan.lukav...@firma.seznam.cz <mailto:jan.lukav...@firma.seznam.cz>>
wrote:
Hi Ravi,
I don't think cgroups will help us, because, we don't want to
impose a hard limit on the memory usage, we just want to allow for
short time periods, when container can consume more memory than
its limit. We don't want to put the limit too high, because that
causes underutilization of our cluster, but setting it
"reasonable" causes applications to fail (because of random
containers being killed because of spikes). That's why we created
the time-window averaging resource calculator, and I was trying to
find out, if anybody else is having similar kind of issues. If so,
I could contribute our extension (and therefore we will not have
to maintain it ourselves in a separate repository :)). The
resource calculator is for hadoop 2.6, and I suppose there might
be larger changes around this in higher versions?
Cheers,
Jan
On 10.8.2016 19:23, Ravi Prakash wrote:
Hi Jan!
Thanks for your explanation. I'm glad that works for you! :-)
https://issues.apache.org/jira/browse/YARN-5202
<https://issues.apache.org/jira/browse/YARN-5202> is something
that Yahoo! talked about at the Hadoop Summit, (and it seems the
community may be going in a similar direction, although not
exactly the same.) There's also
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
<https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java>
. Ideally at my company we'd like memory limits also to be
imposed by Cgroups because we have had the OOM-killer wreak havoc
a couple of times, but from what I know, that is not an option yet.
Cheers
Ravi
On Wed, Aug 10, 2016 at 1:54 AM, Jan Lukavský
<jan.lukav...@firma.seznam.cz
<mailto:jan.lukav...@firma.seznam.cz>> wrote:
Hi Ravi,
we don't run into situation where memory used > RAM, because
memory configured to be used by all containers on a node is
less than the total amount on memory (by a factor of say
10%). The spikes of container memory usage, that are
tolerated due to the averaging don't happen on all containers
at once, but are more of a random nature and therefore mostly
only single running container "spikes", which therefore
doesn't cause any issues. To fully answer your question, we
have overcommit enabled and therefore, if we would run out of
memory, bad things would happen. :) We are aware of that. The
risk of running into OOM-killer can be controlled by the
averaging window length - as the length grows, the more and
more spikes are tolerated. Setting the averaging window
length to 1 switches this feature off, turning it back into
the "standard" behavior, which is why I see it as a extension
of the current approach, which could be interesting to other
people as well.
Jan
On 10.8.2016 02:48, Ravi Prakash wrote:
Hi Jan!
Thanks for your contribution. In your approach what happens
when a few containers on a node are using "excessive" memory
(so that total memory used > RAM available on the machine).
Do you have overcommit enabled?
Thanks
Ravi
On Tue, Aug 9, 2016 at 1:31 AM, Jan Lukavský
<jan.lukav...@firma.seznam.cz
<mailto:jan.lukav...@firma.seznam.cz>> wrote:
Hello community,
I have a question about container resource calculation
in nodemanager. Some time ago a filed JIRA
https://issues.apache.org/jira/browse/YARN-4681
<https://issues.apache.org/jira/browse/YARN-4681>, which
I though might address our problems with container being
killed because of read-only mmaping memory block. The
JIRA has not been resolved yet, but it turned out for
us, that the patch doesn't solve the problem. Some
applications (namely Apache Spark) tend to allocate
really large memory blocks outside JVM heap (using mmap,
but with MAP_PRIVATE), but only for short time periods.
We solved this by creating a smoothing resource
calculator, which averages the memory usage of a
container over some time period (say 5 minutes). This
eliminates the problem of container being killed for
short memory consumption peak, but in the same time
preserves the ability to kill container that *really*
consumes excessive amount of memory.
My question is, does this seem a systematic approach to
you and should I post our patch to the community or am
thinking in a wrong direction from the beginning? :)
Thanks for reactions,
Jan
---------------------------------------------------------------------
To unsubscribe, e-mail:
yarn-dev-unsubscr...@hadoop.apache.org
<mailto:yarn-dev-unsubscr...@hadoop.apache.org>
For additional commands, e-mail:
yarn-dev-h...@hadoop.apache.org
<mailto:yarn-dev-h...@hadoop.apache.org>
--
Jan Lukavský
Vedoucí týmu vývoje
Seznam.cz, a.s.
Radlická 3294/10
15000, Praha 5
jan.lukav...@firma.seznam.cz <mailto:jan.lukav...@firma.seznam.cz>
http://www.seznam.cz
--
Jan Lukavský
Vedoucí týmu vývoje
Seznam.cz, a.s.
Radlická 3294/10
15000, Praha 5
jan.lukav...@firma.seznam.cz
http://www.seznam.cz
