How to redirect the storing of the following files from /tmp to some
other location.
hadoop-os user-namenode.pid
hadoop-os user-datanode.pid
yarn-os user-resourcemanager.pid
yarn-os user-nodemanager.pid
In /tmp, these files are cleared by OS sometime back and I am unable
to shutdown by standard
Hi Susheel,
Adding a new directory to “dfs.datanode.data.dir” will not balance your
disks straightforward. Eventually, by HDFS activity (deleting/invalidating
some block, writing new ones), the disks will become balanced. If you want
to balance them right after adding the new disk and changing
Hi Susheel,
You have to set in your hadoop-env.sh and yarn-env.sh the variabkes:
- HADOOP_PID_DIR
- YARN_PID_DIR
To point to some other directory (most common is /var/run/hadoop-hdfs
hadoop-yarn)
Hope it helps,
Aitor
On 29 September 2014 07:50, Susheel Kumar Gadalay skgada...@gmail.com
wrote:
Hi All,
I want to use custom logging for map-reduce application so i want to
configure my log4j.properties file in hadoop.
I want that logging of all containers should be happen according to my
log4.properties file.
For this i updated different properties in mapred-site.xml configuration
file.
Thanks Aitor.
There is a shell variable HADOOP_PID_DIR refererred in hadoop-env.sh
but not in yarn-env.sh.
On 9/29/14, Aitor Cedres aced...@pivotal.io wrote:
Hi Susheel,
You have to set in your hadoop-env.sh and yarn-env.sh the variabkes:
- HADOOP_PID_DIR
- YARN_PID_DIR
To point to
Thank Aitor.
That is what is my observation too.
I added a new disk location and manually moved some files.
But if 2 locations are given at the beginning itself for
dfs.datanode.data.dir, will hadoop balance the disks usage, if not
perfect because file sizes may differ.
On 9/29/14, Aitor
Hi,
I am using a hadoop map reduce job + mongoDb.
It goes against a data base 252Gb big. During the job the amount of conexions
is over 8000 and we gave already 9Gb RAM. The job is still crashing because of
a OutOfMemory with only a 8% of the mapping done.
Are this numbers normal? Or did we
I think they way it works when HDFS has a list in dfs.datanode.data.dir,
it's basically a round robin between disks. And yes, it may not be perfect
balanced cause of different file sizes.
On 29 September 2014 13:15, Susheel Kumar Gadalay skgada...@gmail.com
wrote:
Thank Aitor.
That is what
Check the file $HADOOP_HOME/bin/yarn-daemon.sh; there is a reference to
YARN_PID_DIR. If it's not set. it will default to /tmp.
On 29 September 2014 13:11, Susheel Kumar Gadalay skgada...@gmail.com
wrote:
Thanks Aitor.
There is a shell variable HADOOP_PID_DIR refererred in hadoop-env.sh
but
I don't have any experience with MongoDB, but just gave my 2 cents here.
Your code is not efficient, as using the += on String, and you could have
reused the Text object in your mapper, as it is a mutable class, to be reused
and avoid creating it again and again like new Text() in the mapper. My
Thanks for your answer.
To your questions:
1. When you claim 96G ram, I am not sure what do you mean?
It is not 96 Gb RAM, it is 9 Gb that our test server has available (is it too
small?).
2. Your code is not efficient, as using the += on String
I need (or at least I don´t have
Hi Takenori,
Thanks for replying but still seem not getting some concepts
I understand that we need to give fs.AbstractFileSystem.s3.impl if we want to
submit job using ./yarn jar with S3 HCFS configured.
But what i don't understand is why 2 interfaces (may be i am novice in HDFS and
hence not
Hi Naga,
But what i don't understand is why 2 interfaces (may be i am novice in
HDFS and hence not able to completely correlate with jira's which you
gave).
A client program is encouraged to use FileContext API instead of FileSystem
API. Here's why
Here are my suggestions originally aims to improve the efficient:
1) In your case, you could use StringBuilder, which has the append method,
should be more efficient to concatenate your string data in this case.2) What I
mean to reuse the Text object is as following: public class mapper
I am currently developing tests that use a mini yarn cluster. Because it is
running on circle-ci I need to use the absolute minimum amount of
memory.
I'm currently setting
conf.setFloat(yarn.
nodemanager.vmem-pmem-ratio, 8.0f);
conf.setBoolean(mapreduce.map.speculative, false);
Thanks
On 9/29/14, Aitor Cedres aced...@pivotal.io wrote:
Check the file $HADOOP_HOME/bin/yarn-daemon.sh; there is a reference to
YARN_PID_DIR. If it's not set. it will default to /tmp.
On 29 September 2014 13:11, Susheel Kumar Gadalay skgada...@gmail.com
wrote:
Thanks Aitor.
There is a
Hi ,
I have 2 different networks in my setup
Job tracker and name node are running in private network,
eclipse client is running on public network
I see the JIRA which is very relevant is there a workaround ?
https://issues.apache.org/jira/browse/HADOOP-1202
--
Warm Regards,
Dear All,
I am not doing load balancing here. I am just copying a file and it is
throwing me an error no space left on the device.
hduser@dn1:~$ df -h
Filesystem Size Used Avail Use%
Mounted on
/dev/xvda2 5.9G 5.1G
Hi folks,
I'm having trouble using HBASE copyTable to seed an existing tables data to
a replication peer. Surely its an oversight in configuration on our part,
but I've scoured the web and doc's for a couple days now.
We have been able to run these jobs with success (perhaps they don't
require
Hi Experts:
I write the following java program to access Kerberosed Hadoop File system:
---
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileStatus;
Resolved: from client machine I need to update the
./etc/hadoop/core-site.xml with site specific info e.g.
property
namehadoop.security.authentication/name
valuekerberos/value
/property
also hdfs-site.xml needs:
property
namedfs.namenode.kerberos.principal/name
Hi Experts:
I can go to oozie console http://oozieserver:11000/oozie/ when our
hadoop cluster has no kerberos setup.
But after Keberos setup, I try the same oozie console from IE, i got error:
type Status report
message
The host name is fully qualified , meaning there is nothing more that I can
add , it just seems the ports might be messed up , but I don't know which
ones
On Mon, Sep 29, 2014 at 12:44 AM, Susheel Kumar Gadalay skgada...@gmail.com
wrote:
I also faced some issue like this.
It shows the URL in
Hi, Matt
Thank you very much for your response!
There were some mistakes in my description as i wrote this mail in a hurry. I
put those properties is in hdfs-site.xml not core-site.xml.
There are four name nodes because i also using HDFS federation, so there are
two nameservices in porperty
You need to start the ZKFC process which will monitor and manage the state of
namenode.
Automatic failover adds two new components to an HDFS deployment: a ZooKeeper
quorum, and the ZKFailoverController process (abbreviated as ZKFC).
Apache ZooKeeper is a highly available service for
25 matches
Mail list logo
