[jira] [Resolved] (HDFS-13397) start-dfs.sh and hdfs --daemon start datanode say "ERROR: Cannot set priority of datanode process XXXX"
[ https://issues.apache.org/jira/browse/HDFS-13397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Hubbs resolved HDFS-13397. --- Resolution: Invalid Release Note: This fix apparently does not work in all cases, will withdraw and re-post after further investigation > start-dfs.sh and hdfs --daemon start datanode say "ERROR: Cannot set priority > of datanode process " > --- > > Key: HDFS-13397 > URL: https://issues.apache.org/jira/browse/HDFS-13397 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs >Affects Versions: 3.0.1 >Reporter: Jeff Hubbs >Priority: Major > > When executing > {code:java} > $HADOOP_HOME/bin/hdfs --daemon start datanode > {code} > as a regular user (e.g. "hdfs") you achieve fail saying > {code:java} > ERROR: Cannot set priority of datanode process > {code} > where is some PID. > It turned out that this is because at least on Gentoo Linux (and I think this > is pretty well universal), by default a regular user process can't increase > the priority of itself or any of the user's other processes. To fix this, I > added these lines to /etc/security/limits.conf [NOTE: the users hdfs, yarn, > and mapred are in the group called hadoop on this system]: > {code:java} > @hadoop hard nice -15 > @hadoop hard priority -15 > {code} > This change will need to be made on all datanodes. > The need to enable [at minimum] the hdfs user to raise its processes' > priority needs to be added to the documentation. This is not a problem I > observed under 3.0.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13397) start-dfs.sh and hdfs --daemon start datanode say "ERROR: Cannot set priority of datanode process XXXX"
Jeff Hubbs created HDFS-13397: - Summary: start-dfs.sh and hdfs --daemon start datanode say "ERROR: Cannot set priority of datanode process " Key: HDFS-13397 URL: https://issues.apache.org/jira/browse/HDFS-13397 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs Affects Versions: 3.0.1 Reporter: Jeff Hubbs When executing {code:java} $HADOOP_HOME/bin/hdfs --daemon start datanode {code} as a regular user (e.g. "hdfs") you achieve fail saying {code:java} ERROR: Cannot set priority of datanode process {code} where is some PID. It turned out that this is because at least on Gentoo Linux (and I think this is pretty well universal), by default a regular user process can't increase the priority of itself or any of the user's other processes. To fix this, I added these lines to /etc/security/limits.conf [NOTE: the users hdfs, yarn, and mapred are in the group called hadoop on this system]: {code:java} @hadoop hard nice -15 @hadoop hard priority -15 {code} This change will need to be made on all datanodes. The need to enable [at minimum] the hdfs user to raise its processes' priority needs to be added to the documentation. This is not a problem I observed under 3.0.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13396) start-dfs.sh and stop-dfs.sh has malformed command; doesn't use workers file
Jeff Hubbs created HDFS-13396: - Summary: start-dfs.sh and stop-dfs.sh has malformed command; doesn't use workers file Key: HDFS-13396 URL: https://issues.apache.org/jira/browse/HDFS-13396 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Affects Versions: 3.0.1, 3.0.0 Environment: Hadoop 3.0.1 binary distribution on Gentoo Linux, Icedtea JRE Reporter: Jeff Hubbs In 3.0.1's start-dfs.sh, the command to start the datanodes reads as follows: {code:java} hadoop_uservar_su hdfs datanode "${HADOOP_HDFS_HOME}/bin/hdfs" \ --workers \ --config "${HADOOP_CONF_DIR}" \ --daemon start \ datanode ${dataStartOpt} {code} This doesn't work; executing the script produces this: {code:java} hdfs@msba02a ~ $ $HADOOP_HOME/sbin/start-dfs.sh Starting namenodes on [msba02a.bus.emory.ddns] Starting datanodes ^/opt/hadoop/3: ssh: Could not resolve hostname ^/opt/hadoop/3.0.1/etc/hadoop/workers: Name or service not known pdsh@msba02a: ^/opt/hadoop/3: ssh exited with exit code 255 Starting secondary namenodes [msba02a] {code} It misinterprets the value of HADOOP_CONF_DIR as one of the names of a machine it is supposed to access. The workaround I developed involves the --hostnames option like so, changing the one-name-per-line workers file into a comma-separated list: {code:java} hadoop_uservar_su hdfs datanode "${HADOOP_HDFS_HOME}/bin/hdfs" \ --workers \ --hostnames `sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/,/g' ${HADOOP_CONF_DIR}/workers` \ --config "${HADOOP_CONF_DIR}" \ --daemon start \ datanode ${dataStartOpt} {code} A similar change had to be made to stop-dfs.sh. I've verified that HADOOP_HDFS_HOME and HADOOP_CONF_DIR are set correctly within the script at the point where this command executes. This problem also exists in start-dfs.sh/stop-dfs.sh in 3.0.0, although the original invocation differs slightly from 3.0.1. In 3.0.1, I'm running into another problem with getting datanodes started (was fine in 3.0.0) but I couldn't hit that problem until I got past this one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org