Siddharth Wagle created AMBARI-3074:
---------------------------------------
Summary: Ambari wont start NodeManager because one of multiple
folders not created
Key: AMBARI-3074
URL: https://issues.apache.org/jira/browse/AMBARI-3074
Project: Ambari
Issue Type: Bug
Components: agent
Affects Versions: 1.4.0
Reporter: Siddharth Wagle
Assignee: Siddharth Wagle
Fix For: 1.4.1
{{yarn-site}} having:
{noformat}
"yarn.nodemanager.local-dirs" :
"/grid/0/hadoop/yarn,/grid/1/hadoop/yarn,/grid/2/hadoop/yarn,/grid/3/hadoop/yarn,/grid/4/hadoop/yarn,/grid/5/hadoop/yarn",
"yarn.nodemanager.log-dirs" :
"/grid/0/hadoop/yarn,/grid/1/hadoop/yarn,/grid/2/hadoop/yarn,/grid/3/hadoop/yarn,/grid/4/hadoop/yarn,/grid/5/hadoop/yarn",
{noformat}
Now {{/grid/3}} was mounted as read-only due to some disk errors. Though other
folders got successfully created, Ambari will not start the NodeManager process.
{noformat}
notice:
/Stage[1]/Hdp::Snappy::Package/Hdp::Snappy::Package::Ln[32]/Hdp::Exec[hdp::snappy::package::ln
32]/Exec[hdp::snappy::package::ln 32]/returns: executed successfully
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir
-p /grid/3/hadoop/yarn]/Exec[mkdir -p /grid/3/hadoop/yarn]/returns: mkdir:
cannot create directory `/grid/3/hadoop': Read-only file system
err:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir
-p /grid/3/hadoop/yarn]/Exec[mkdir -p /grid/3/hadoop/yarn]/returns: change
from notrun to 0 failed: mkdir -p /grid/3/hadoop/yarn returned 1 instead of one
of [0] at /var/lib/ambari-agent/puppet/modules/hdp/manifests/init.pp:479
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir
-p /grid/3/hadoop/yarn]/Anchor[hdp::exec::mkdir -p /grid/3/hadoop/yarn::end]:
Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir
-p /grid/3/hadoop/yarn]/Anchor[hdp::exec::mkdir -p /grid/3/hadoop/yarn::end]:
Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Directory[/grid/3/hadoop/yarn]/File[/grid/3/hadoop/yarn]:
Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Directory[/grid/3/hadoop/yarn]/File[/grid/3/hadoop/yarn]:
Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[capacity-scheduler]/File[/etc/hadoop/conf/capacity-scheduler.xml]/content:
content changed '{md5}e5d17c21c7a5e1db9f3af35cba71df0a' to
'{md5}2ca1d267a46f1aecac726caabaa16774'
notice:
/Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[capacity-scheduler]/File[/etc/hadoop/conf/capacity-scheduler.xml]/owner:
owner changed 'hdfs' to 'yarn'
notice:
/Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[core-site]/File[/etc/hadoop/conf/core-site.xml]/content:
content changed '{md5}86d742a780d59a957ea0a283dec03784' to
'{md5}8506e4402ba8140ea4f9fed97b6f94e2'
notice:
/Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[yarn-site]/File[/etc/hadoop/conf/yarn-site.xml]/content:
content changed '{md5}d84a967ce47a6b77734ed8f53d817c6e' to
'{md5}42940cca6e8f64ae5de50524fb131274'
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::begin]:
Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::begin]:
Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir
-p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p
/var/log/hadoop-yarn::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has
failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir
-p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p
/var/log/hadoop-yarn::begin]: Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir
-p /var/log/hadoop-yarn]/Exec[mkdir -p /var/log/hadoop-yarn]: Dependency
Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir
-p /var/log/hadoop-yarn]/Exec[mkdir -p /var/log/hadoop-yarn]: Skipping because
of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir
-p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p
/var/log/hadoop-yarn::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has
failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir
-p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p
/var/log/hadoop-yarn::end]: Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Directory[/var/log/hadoop-yarn]/File[/var/log/hadoop-yarn]:
Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Directory[/var/log/hadoop-yarn]/File[/var/log/hadoop-yarn]:
Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir
-p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p
/var/run/hadoop-yarn/yarn::begin]: Dependency Exec[mkdir -p
/grid/3/hadoop/yarn] has failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir
-p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p
/var/run/hadoop-yarn/yarn::begin]: Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir
-p /var/run/hadoop-yarn/yarn]/Exec[mkdir -p /var/run/hadoop-yarn/yarn]:
Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir
-p /var/run/hadoop-yarn/yarn]/Exec[mkdir -p /var/run/hadoop-yarn/yarn]:
Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir
-p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p
/var/run/hadoop-yarn/yarn::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn]
has failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir
-p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p
/var/run/hadoop-yarn/yarn::end]: Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Directory[/var/run/hadoop-yarn/yarn]/File[/var/run/hadoop-yarn/yarn]:
Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Directory[/var/run/hadoop-yarn/yarn]/File[/var/run/hadoop-yarn/yarn]:
Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su -
yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
nodemanager']/Anchor[hdp::exec::su - yarn -c 'export
HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
nodemanager'::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has
failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su -
yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
nodemanager']/Anchor[hdp::exec::su - yarn -c 'export
HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
nodemanager'::begin]: Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su -
yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
nodemanager']/Exec[su - yarn -c 'export
HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
nodemanager']: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su -
yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
nodemanager']/Exec[su - yarn -c 'export
HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
nodemanager']: Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su -
yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
nodemanager']/Anchor[hdp::exec::su - yarn -c 'export
HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
nodemanager'::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures:
true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su -
yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
nodemanager']/Anchor[hdp::exec::su - yarn -c 'export
HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec &&
/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start
nodemanager'::end]: Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep
5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps
`cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
2>&1]/Anchor[hdp::exec::sleep 5; ls
/var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat
/var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1::begin]:
Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep
5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps
`cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
2>&1]/Anchor[hdp::exec::sleep 5; ls
/var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat
/var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1::begin]:
Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep
5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps
`cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
2>&1]/Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid
>/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid`
>/dev/null 2>&1]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures:
true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep
5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps
`cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
2>&1]/Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid
>/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid`
>/dev/null 2>&1]: Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep
5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps
`cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
2>&1]/Anchor[hdp::exec::sleep 5; ls
/var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat
/var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1::end]:
Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep
5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps
`cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null
2>&1]/Anchor[hdp::exec::sleep 5; ls
/var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat
/var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1::end]:
Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::end]:
Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
warning:
/Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::end]:
Skipping because of failed dependencies
notice: /Stage[2]/Hdp-yarn::Nodemanager/Anchor[hdp-yarn::nodemanager::end]:
Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
warning: /Stage[2]/Hdp-yarn::Nodemanager/Anchor[hdp-yarn::nodemanager::end]:
Skipping because of failed dependencies
notice:
/Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[mapred-site]/File[/etc/hadoop/conf/mapred-site.xml]/content:
content changed '{md5}093cb1899b3c3b9dc4a7c1c93729c18b' to
'{md5}4c462999cc47e6f6ba0e6381d71d81ba'
notice:
/Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[mapred-site]/File[/etc/hadoop/conf/mapred-site.xml]/owner:
owner changed 'mapred' to 'yarn'
notice: Finished catalog run in 2.39 seconds
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
