AW: AW: Hadoop 2.6.0 - No DataNode to stop

[Daniel Klinger]

Hi,

 

thanks for your help. The HADOOP_PID_DIR variable is pointing to 
/var/run/cluster/hadoop (which has hdfs:hadoop) as it’s owner. 3 PID are 
created there (datanode namenode and secure_dn). It looks like the PID was 
written but there was a readproblem.

 

I did chmod –R 777 on the folder and now the Datanodes are Stopped correctly. 
It only works when I’m running the start and stop command as user HDFS. If I 
try to start and stop as root (like its documented in the Documentation I still 
get the “no Datanode to stop” error.

 

Is it important to start the DN as root? The only thing I recognized is the 
secure_dn PID-File is not created when im starting the Datanode as HDFS-User. 
Is this a Problem?

 

Greets

DK

Von: Ulul [mailto:had...@ulul.org] 
Gesendet: Montag, 2. März 2015 21:50
An: user@hadoop.apache.org
Betreff: Re: AW: Hadoop 2.6.0 - No DataNode to stop

 

Hi
The hadoop-daemon.sh script prints the no $command to stop if it doesn'f find 
the pid file.
You should echo the $pid variable and see if you hava a correct pid file there.
Ulul

Le 02/03/2015 13:53, Daniel Klinger a écrit :

Thanks for your help. But unfortunatly this didn’t do the job. Here’s the 
Shellscript I’ve written to start my cluster (the scripts on the other node 
only contains the command to start the datanode respectively the command to 
start the Nodemanager on the other node (with the right user (hdfs / yarn)):

 

 

#!/bin/bash

# Start 
HDFS-

# Start Namenode

su - hdfs -c "$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR 
--script hdfs start namenode"

wait

 

# Start all Datanodes

export HADOOP_SECURE_DN_USER=hdfs

su - hdfs -c "$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR 
--script hdfs start datanode"

wait

ssh root@hadoop-data.klinger.local   
'bash startDatanode.sh'

wait

 

# Start Resourcemanager

su - yarn -c "$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR 
start resourcemanager"

wait

 

# Start Nodemanager on all Nodes

su - yarn -c "$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR 
start nodemanager"

wait

ssh root@hadoop-data.klinger.local   
'bash startNodemanager.sh'

wait

 

# Start Proxyserver

#su - yarn -c "$HADOOP_YARN_HOME/bin/yarn start proxyserver --config 
$HADOOP_CONF_DIR"

#wait

 

# Start Historyserver

su - mapred -c "$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver 
--config $HADOOP_CONF_DIR"

wait

 

This script generates the following output:

 

starting namenode, logging to 
/var/log/cluster/hadoop/hadoop-hdfs-namenode-hadoop.klinger.local.out

starting datanode, logging to 
/var/log/cluster/hadoop/hadoop-hdfs-datanode-hadoop.klinger.local.out

starting datanode, logging to 
/var/log/cluster/hadoop/hadoop-hdfs-datanode-hadoop-data.klinger.local.out

starting resourcemanager, logging to 
/var/log/cluster/yarn/yarn-yarn-resourcemanager-hadoop.klinger.local.out

starting nodemanager, logging to 
/var/log/cluster/yarn/yarn-yarn-nodemanager-hadoop.klinger.local.out

starting nodemanager, logging to 
/var/log/cluster/yarn/yarn-yarn-nodemanager-hadoop-data.klinger.local.out

starting historyserver, logging to 
/var/log/cluster/mapred/mapred-mapred-historyserver-hadoop.klinger.local.out

 

Following my stopscript and it’s output:

 

#!/bin/bash

# Stop 
HDFS

# Stop Namenode

su - hdfs -c "$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR 
--script hdfs stop namenode"

 

# Stop all Datanodes

su - hdfs -c "$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR 
--script hdfs stop datanode"

ssh root@hadoop-data.klinger.local   
'bash stopDatanode.sh'

 

# Stop Resourcemanager

su - yarn -c "$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR 
stop resourcemanager"

 

#Stop Nodemanager on all Hosts

su - yarn -c "$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR 
stop nodemanager"

ssh root@hadoop-data.klinger.local   
'bash stopNodemanager.sh'

 

#Stop Proxyserver

#su - yarn -c "$HADOOP_YARN_HOME/bin/yarn stop proxyserver --config 
$HADOOP_CONF_DIR"

 

#Stop Historyserver

su - mapred -c "$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver 
--config $HADOOP_CONF_DIR"

 

stopping namenode

no datanode to stop

no datanode to stop

stopping resourcemanager

stopping nodemanager

stopping nodemanager

nodemanager did not stop gracefully after 5 seconds: killing with kill -9

stopping historyserver

 

Is there may be anything wrong with my commands?

 

Greets

DK

 

Von: Varun Kumar [mailto:varun@gmail.com] 
Gesendet: Montag, 2. März 2015 05:28
An: user
Betreff: Re: 

Re: AW: AW: Hadoop 2.6.0 - No DataNode to stop

[Ulul]

Hi

As a general rule, you should never run an applicative daemon as root 
since any vulnerabilty can allow a malicious intruder to get full 
control of the system.

The documentation does not advise to start hadoop as root :
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Hadoop_Startup
will show you hadoop being started as a regular user

I'm puzzled by the fact that root would be barred from accessing a file. 
The only case I can think of would be an nfs mount with root squashing.


And there should be no need to use the 777 bypass as long as you're 
using the same user to start and stop your daemons.


Ulul

Le 03/03/2015 00:14, Daniel Klinger a écrit :


Hi,

thanks for your help. The HADOOP_PID_DIR variable is pointing to 
/var/run/cluster/hadoop (which has hdfs:hadoop) as it’s owner. 3 PID 
are created there (datanode namenode and secure_dn). It looks like the 
PID was written but there was a readproblem.


I did chmod –R 777 on the folder and now the Datanodes are Stopped 
correctly. It only works when I’m running the start and stop command 
as user HDFS. If I try to start and stop as root (like its documented 
in the Documentation I still get the “no Datanode to stop” error.


Is it important to start the DN as root? The only thing I recognized 
is the secure_dn PID-File is not created when im starting the Datanode 
as HDFS-User. Is this a Problem?


Greets

DK

*Von:*Ulul [mailto:had...@ulul.org]
*Gesendet:* Montag, 2. März 2015 21:50
*An:* user@hadoop.apache.org
*Betreff:* Re: AW: Hadoop 2.6.0 - No DataNode to stop

Hi
The hadoop-daemon.sh script prints the no $command to stop if it 
doesn'f find the pid file.
You should echo the $pid variable and see if you hava a correct pid 
file there.

Ulul

Le 02/03/2015 13:53, Daniel Klinger a écrit :

Thanks for your help. But unfortunatly this didn’t do the job.
Here’s the Shellscript I’ve written to start my cluster (the
scripts on the other node only contains the command to start the
datanode respectively the command to start the Nodemanager on the
other node (with the right user (hdfs / yarn)):

#!/bin/bash

# Start

HDFS-

# Start Namenode

su - hdfs -c "$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config
$HADOOP_CONF_DIR --script hdfs start namenode"

wait

# Start all Datanodes

export HADOOP_SECURE_DN_USER=hdfs

su - hdfs -c "$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config
$HADOOP_CONF_DIR --script hdfs start datanode"

wait

ssh root@hadoop-data.klinger.local
 'bash startDatanode.sh'

wait

# Start Resourcemanager

su - yarn -c "$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config
$HADOOP_CONF_DIR start resourcemanager"

wait

# Start Nodemanager on all Nodes

su - yarn -c "$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config
$HADOOP_CONF_DIR start nodemanager"

wait

ssh root@hadoop-data.klinger.local
 'bash startNodemanager.sh'

wait

# Start Proxyserver

#su - yarn -c "$HADOOP_YARN_HOME/bin/yarn start proxyserver
--config $HADOOP_CONF_DIR"

#wait

# Start Historyserver

su - mapred -c "$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start
historyserver --config $HADOOP_CONF_DIR"

wait

This script generates the following output:

starting namenode, logging to
/var/log/cluster/hadoop/hadoop-hdfs-namenode-hadoop.klinger.local.out

starting datanode, logging to
/var/log/cluster/hadoop/hadoop-hdfs-datanode-hadoop.klinger.local.out

starting datanode, logging to
/var/log/cluster/hadoop/hadoop-hdfs-datanode-hadoop-data.klinger.local.out

starting resourcemanager, logging to
/var/log/cluster/yarn/yarn-yarn-resourcemanager-hadoop.klinger.local.out

starting nodemanager, logging to
/var/log/cluster/yarn/yarn-yarn-nodemanager-hadoop.klinger.local.out

starting nodemanager, logging to
/var/log/cluster/yarn/yarn-yarn-nodemanager-hadoop-data.klinger.local.out

starting historyserver, logging to
/var/log/cluster/mapred/mapred-mapred-historyserver-hadoop.klinger.local.out

Following my stopscript and it’s output:

#!/bin/bash

# Stop

HDFS

# Stop Namenode

su - hdfs -c "$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config
$HADOOP_CONF_DIR --script hdfs stop namenode"

# Stop all Datanodes

su - hdfs -c "$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config
$HADOOP_CONF_DIR --script hdfs stop datanode"

ssh root@hadoop-data.klinger.local
 'bash stopDatanode.sh'

# Stop Resourcemanager

su - yarn -c "$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config
$HADOOP_C