[jira] [Resolved] (HDFS-13397) start-dfs.sh and hdfs --daemon start datanode say "ERROR: Cannot set priority of datanode process XXXX"
[
https://issues.apache.org/jira/browse/HDFS-13397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Hubbs resolved HDFS-13397.
---
Resolution: Invalid
Release Note: This fix apparently does not work in all cases, will withdraw
and re-post after further investigation
> start-dfs.sh and hdfs --daemon start datanode say "ERROR: Cannot set priority
> of datanode process "
> ---
>
> Key: HDFS-13397
> URL: https://issues.apache.org/jira/browse/HDFS-13397
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: hdfs
>Affects Versions: 3.0.1
>Reporter: Jeff Hubbs
>Priority: Major
>
> When executing
> {code:java}
> $HADOOP_HOME/bin/hdfs --daemon start datanode
> {code}
> as a regular user (e.g. "hdfs") you achieve fail saying
> {code:java}
> ERROR: Cannot set priority of datanode process
> {code}
> where is some PID.
> It turned out that this is because at least on Gentoo Linux (and I think this
> is pretty well universal), by default a regular user process can't increase
> the priority of itself or any of the user's other processes. To fix this, I
> added these lines to /etc/security/limits.conf [NOTE: the users hdfs, yarn,
> and mapred are in the group called hadoop on this system]:
> {code:java}
> @hadoop hard nice -15
> @hadoop hard priority -15
> {code}
> This change will need to be made on all datanodes.
> The need to enable [at minimum] the hdfs user to raise its processes'
> priority needs to be added to the documentation. This is not a problem I
> observed under 3.0.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13397) start-dfs.sh and hdfs --daemon start datanode say "ERROR: Cannot set priority of datanode process XXXX"
Jeff Hubbs created HDFS-13397:
-
Summary: start-dfs.sh and hdfs --daemon start datanode say "ERROR:
Cannot set priority of datanode process "
Key: HDFS-13397
URL: https://issues.apache.org/jira/browse/HDFS-13397
Project: Hadoop HDFS
Issue Type: New Feature
Components: hdfs
Affects Versions: 3.0.1
Reporter: Jeff Hubbs
When executing
{code:java}
$HADOOP_HOME/bin/hdfs --daemon start datanode
{code}
as a regular user (e.g. "hdfs") you achieve fail saying
{code:java}
ERROR: Cannot set priority of datanode process
{code}
where is some PID.
It turned out that this is because at least on Gentoo Linux (and I think this
is pretty well universal), by default a regular user process can't increase the
priority of itself or any of the user's other processes. To fix this, I added
these lines to /etc/security/limits.conf [NOTE: the users hdfs, yarn, and
mapred are in the group called hadoop on this system]:
{code:java}
@hadoop hard nice -15
@hadoop hard priority -15
{code}
This change will need to be made on all datanodes.
The need to enable [at minimum] the hdfs user to raise its processes' priority
needs to be added to the documentation. This is not a problem I observed under
3.0.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13396) start-dfs.sh and stop-dfs.sh has malformed command; doesn't use workers file
Jeff Hubbs created HDFS-13396:
-
Summary: start-dfs.sh and stop-dfs.sh has malformed command;
doesn't use workers file
Key: HDFS-13396
URL: https://issues.apache.org/jira/browse/HDFS-13396
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Affects Versions: 3.0.1, 3.0.0
Environment: Hadoop 3.0.1 binary distribution on Gentoo Linux, Icedtea
JRE
Reporter: Jeff Hubbs
In 3.0.1's start-dfs.sh, the command to start the datanodes reads as follows:
{code:java}
hadoop_uservar_su hdfs datanode "${HADOOP_HDFS_HOME}/bin/hdfs" \
--workers \
--config "${HADOOP_CONF_DIR}" \
--daemon start \
datanode ${dataStartOpt}
{code}
This doesn't work; executing the script produces this:
{code:java}
hdfs@msba02a ~ $ $HADOOP_HOME/sbin/start-dfs.sh
Starting namenodes on [msba02a.bus.emory.ddns]
Starting datanodes
^/opt/hadoop/3: ssh: Could not resolve hostname
^/opt/hadoop/3.0.1/etc/hadoop/workers: Name or service not known
pdsh@msba02a: ^/opt/hadoop/3: ssh exited with exit code 255
Starting secondary namenodes [msba02a]
{code}
It misinterprets the value of HADOOP_CONF_DIR as one of the names of a machine
it is supposed to access.
The workaround I developed involves the --hostnames option like so, changing
the one-name-per-line workers file into a comma-separated list:
{code:java}
hadoop_uservar_su hdfs datanode "${HADOOP_HDFS_HOME}/bin/hdfs" \
--workers \
--hostnames `sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/,/g'
${HADOOP_CONF_DIR}/workers` \
--config "${HADOOP_CONF_DIR}" \
--daemon start \
datanode ${dataStartOpt}
{code}
A similar change had to be made to stop-dfs.sh. I've verified that
HADOOP_HDFS_HOME and HADOOP_CONF_DIR are set correctly within the script at the
point where this command executes.
This problem also exists in start-dfs.sh/stop-dfs.sh in 3.0.0, although the
original invocation differs slightly from 3.0.1.
In 3.0.1, I'm running into another problem with getting datanodes started (was
fine in 3.0.0) but I couldn't hit that problem until I got past this one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
