-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53248/
-----------------------------------------------------------
(Updated Oct. 27, 2016, 11:20 p.m.)
Review request for Ambari and Jaimin Jetly.
Repository: ambari
Description
-------
As HDFS-11047 reported, DirectoryScanner does scan by deep copying
FinalizedReplica. In a deployment with 500,000+ blocks, we've seen the DN heap
usage being accumulated to high peaks very quickly. Deep copies of
FinalizedReplica will make DN heap usage even worse if directory scans are
scheduled more frequently.
Another factor is that huge number of ScanInfo instances corresponding to HDFS
blocks are lingering in garbage to eat many heap memories until a full GC takes
place.
This proposes adding JVM settings to force GC more frequently to release
DataNode heap consumed as a result of two aforementioned reasons, i.e. add the
options to HADOOP_DATANODE_OPTS
{noformat}
-XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly
-XX:ConcGCThreads=8 -XX:+UseConcMarkSweepGC
{noformat}
Diffs
-----
ambari-funtest/src/test/resources/stacks/HDP/0.2/services/HDFS/configuration/hadoop-env.xml
2f88e06
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml
32442d9
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/configuration/hadoop-env.xml
5188913
ambari-server/src/main/resources/stacks/HDP/2.0.6.GlusterFS/services/GLUSTERFS/configuration/hadoop-env.xml
5c1c441
ambari-server/src/main/resources/stacks/HDP/2.1.GlusterFS/services/GLUSTERFS/configuration/hadoop-env.xml
95fc12a
ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml
7b92a6c
ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/GLUSTERFS/configuration/hadoop-env.xml
f005a75
ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/configuration/hadoop-env.xml
0371194
ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/HDFS/configuration/hadoop-env.xml
08a1847
ambari-server/src/test/resources/stacks/HDP/0.2/services/HDFS/configuration/hadoop-env.xml
228b2e0
ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/HDFS/configuration/hadoop-env.xml
b8b5f8c
ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hadoop-env.xml
2f9b95f
Diff: https://reviews.apache.org/r/53248/diff/
Testing
-------
Thanks,
XIAOBING ZHOU
