-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25265/
-----------------------------------------------------------
Review request for Ambari and Dmitro Lisnichenko.
Bugs: AMBARI-7119
https://issues.apache.org/jira/browse/AMBARI-7119
Repository: ambari
Description
-------
PROBLEM: log4j settings made via Ambari update the log4j file but do not take
any affect when restarting HDFS. It seems there are hardcoded settings in
/usr/lib/hadoop/libexec/hadoop-config.sh such as this at line 221:
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.root.logger=$
{HADOOP_ROOT_LOGGER:-INFO,console}
"
BUSINESS IMPACT: Customers have to change core files or set environment
variables explicitly by setting up a profile script
STEPS TO REPRODUCE: Log in to Ambari and change log4j properties such that
hadoop.root.logger=INFO,DRFA. The log4j file is updated in /etc/hadoop/conf.
Restart the HDFS service. Do a ps -ef | grep <PID> for the namenode. The
process shows duplicate entries for several properties and does not show the
logging change. Here is the duplication and incorrect root logger setting seen
locally in testing:
hdfs 4304 1 14 07:26 ? 00:00:10 /usr/jdk64/jdk1.7.0_45/bin/java
-Dproc_namenode -Xmx1024m -Djava.net.preferIPv4Stack=true
-Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true
-Dhadoop.log.dir=/var/log/hadoop/hdfs -Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
-Dhadoop.root.logger=INFO,console
-Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-
amd64-64:/usr/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml
-Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/hdfs
-Dhadoop.log.file=hadoop-hdfs-namenode-sandbox.hortonworks.com.log
-Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
-Dhadoop.root.logger=INFO,RFA -Djava.library.path=:/usr/lib/hadoop/lib/native
/Linux-amd64-64:/usr/lib/hadoop/lib/native:/usr/lib/hadoop/lib/native/Linux-
amd64-64:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -server
-XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC
-XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:NewSize=100m
-XX:MaxNewSize=50m -Xloggc:/var/log/hadoop/hdfs/gc.log-201407140726
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
-Xms1024m -Xmx1024m -Dhadoop.security.logger=INFO,DRFAS
-Dhdfs.audit.logger=INFO,DRFAAUDIT -server -XX:ParallelGCThreads=8
-XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log
-XX:NewSize=100m -XX:MaxNewSize=50m
-Xloggc:/var/log/hadoop/hdfs/gc.log-201407140726 -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xms1024m
-Xmx1024m -Dhadoop.security.logger=INFO,DRFAS
-Dhdfs.audit.logger=INFO,DRFAAUDIT -server -XX:ParallelGCThreads=8
-XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log
-XX:NewSize=100m -XX:MaxNewSize=50m
-Xloggc:/var/log/hadoop/hdfs/gc.log-201407140726 -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xms1024m
-Xmx1024m -Dhadoop.security.logger=INFO,DRFA
ACTUAL BEHAVIOR: log4j changes made in Ambari do not persist in process. It
seems there are values set in /usr/lib/hadoop/libexec/hadoop-config.sh that
override no matter what. There is also duplication of settings, assuming this
is from hadoop-config.sh as well.
EXPECTED BEHAVIOR: Settings made in Ambari should be persisted and used by the
process.
SUPPORT ANALYSIS: Support made changes to log4j in Ambari on a test cluster
and they were not used
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog170.java
0ac9da7
ambari-server/src/main/resources/stacks/HDP/1.3.2/hooks/after-INSTALL/scripts/params.py
a4fb172
ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HDFS/configuration/hadoop-env.xml
ba704ba
ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HDFS/configuration/hdfs-log4j.xml
1366c5d
ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HDFS/package/scripts/params.py
58f01f8
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
3fe5905
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/configuration/hadoop-env.xml
573a67c
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/configuration/hdfs-log4j.xml
976a3c2
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/package/scripts/params.py
5c2f792
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog170Test.java
e2ce75e
ambari-web/app/data/HDP2/site_properties.js 3ba0246
ambari-web/app/data/site_properties.js 1f6bb63
Diff: https://reviews.apache.org/r/25265/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
