Hadoop shutdown scripts failing
How to redirect the storing of the following files from /tmp to some other location. hadoop-os user-namenode.pid hadoop-os user-datanode.pid yarn-os user-resourcemanager.pid yarn-os user-nodemanager.pid In /tmp, these files are cleared by OS sometime back and I am unable to shutdown by standard scripts stop-dfs.sh or stop-yarn.sh In core-site.xml I have given property hadoop.tmp.dir but still these pid files are present in /tmp only,
Re: No space when running a hadoop job
Hi Susheel, Adding a new directory to “dfs.datanode.data.dir” will not balance your disks straightforward. Eventually, by HDFS activity (deleting/invalidating some block, writing new ones), the disks will become balanced. If you want to balance them right after adding the new disk and changing the “dfs.datanode.data.dir” value, you have to shutdown the DN and manually move (mv) some files in the old directory to the new one. The balancer will try to balance the usage between HDFS nodes, but it won't care about internal node disks utilization. For your particular case, the balancer won't fix your issue. Hope it helps, Aitor On 29 September 2014 05:53, Susheel Kumar Gadalay skgada...@gmail.com wrote: You mean if multiple directory locations are given, Hadoop will balance the distribution of files across these different directories. But normally we start with 1 directory location and once it is reaching the maximum, we add new directory. In this case how can we balance the distribution of files? One way is to list the files and move. Will start balance script will work? On 9/27/14, Alexander Pivovarov apivova...@gmail.com wrote: It can read/write in parallel to all drives. More hdd more io speed. On Sep 27, 2014 7:28 AM, Susheel Kumar Gadalay skgada...@gmail.com wrote: Correct me if I am wrong. Adding multiple directories will not balance the files distributions across these locations. Hadoop will add exhaust the first directory and then start using the next, next .. How can I tell Hadoop to evenly balance across these directories. On 9/26/14, Matt Narrell matt.narr...@gmail.com wrote: You can add a comma separated list of paths to the “dfs.datanode.data.dir” property in your hdfs-site.xml mn On Sep 26, 2014, at 8:37 AM, Abdul Navaz navaz@gmail.com wrote: Hi I am facing some space issue when I saving file into HDFS and/or running map reduce job. root@nn:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda2 5.9G 5.9G 0 100% / udev 98M 4.0K 98M 1% /dev tmpfs 48M 192K 48M 1% /run none 5.0M 0 5.0M 0% /run/lock none 120M 0 120M 0% /run/shm overflow 1.0M 4.0K 1020K 1% /tmp /dev/xvda4 7.9G 147M 7.4G 2% /mnt 172.17.253.254:/q/groups/ch-geni-net/Hadoop-NET 198G 108G 75G 59% /groups/ch-geni-net/Hadoop-NET 172.17.253.254:/q/proj/ch-geni-net 198G 108G 75G 59% /proj/ch-geni-net root@nn:~# I can see there is no space left on /dev/xvda2. How can I make hadoop to see newly mounted /dev/xvda4 ? Or do I need to move the file manually from /dev/xvda2 to xvda4 ? Thanks Regards, Abdul Navaz Research Assistant University of Houston Main Campus, Houston TX Ph: 281-685-0388
Re: Hadoop shutdown scripts failing
Hi Susheel, You have to set in your hadoop-env.sh and yarn-env.sh the variabkes: - HADOOP_PID_DIR - YARN_PID_DIR To point to some other directory (most common is /var/run/hadoop-hdfs hadoop-yarn) Hope it helps, Aitor On 29 September 2014 07:50, Susheel Kumar Gadalay skgada...@gmail.com wrote: How to redirect the storing of the following files from /tmp to some other location. hadoop-os user-namenode.pid hadoop-os user-datanode.pid yarn-os user-resourcemanager.pid yarn-os user-nodemanager.pid In /tmp, these files are cleared by OS sometime back and I am unable to shutdown by standard scripts stop-dfs.sh or stop-yarn.sh In core-site.xml I have given property hadoop.tmp.dir but still these pid files are present in /tmp only,
How to overwrite container-log4j.properties file
Hi All, I want to use custom logging for map-reduce application so i want to configure my log4j.properties file in hadoop. I want that logging of all containers should be happen according to my log4.properties file. For this i updated different properties in mapred-site.xml configuration file. For example: property namemapreduce.map.log.level/name valueINFO,flume -Dlog4j.configuration=/home/hanish/Desktop/log4j.properties/value /property property namemapreduce.reduce.log.level/name valueINFO,flume -Dlog4j.configuration=/home/hanish/Desktop/log4j.properties -Dhadoop.root.logger=INFO,flume/value /property property nameyarn.app.mapreduce.am.command-opts/name value-Dlog4j.configuration=/home/hanish/Desktop/log4j.properties -Xmx1024m -Dhadoop.root.logger=INFO,flume/value /property property nameyarn.app.mapreduce.am.admin-command-opts/name value-Dlog4j.configuration=/home/hanish/Desktop/log4j.properties/value /property Using this configuration MRAppMaster is getting started with more one log4j.configuration variables and logging is done according to container-log4j.properties file. Please find the jps information of Application master is: 19812 MRAppMaster -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hanish/opt/hadoop-2.2.0/logs/userlogs/application_1411987111980_0001/container_1411987111980_0001_01_01 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dlog4j.configuration=/home/hanish/Desktop/log4j.properties -Dlog4j.configuration=/home/hanish/Desktop/log4j.properties -Xmx1024m -Dhadoop.root.logger=INFO,flume 5959 Application -Xmx20m -Dflume.root.logger=DEBUG,console -Djava.library.path=:/home/hanish/opt/hadoop-2.2.0/lib/native Please let me know how to overwrite container-log4j.properties file ? -- *Thanks Regards* *Hanish Bansal*
Re: Hadoop shutdown scripts failing
Thanks Aitor. There is a shell variable HADOOP_PID_DIR refererred in hadoop-env.sh but not in yarn-env.sh. On 9/29/14, Aitor Cedres aced...@pivotal.io wrote: Hi Susheel, You have to set in your hadoop-env.sh and yarn-env.sh the variabkes: - HADOOP_PID_DIR - YARN_PID_DIR To point to some other directory (most common is /var/run/hadoop-hdfs hadoop-yarn) Hope it helps, Aitor On 29 September 2014 07:50, Susheel Kumar Gadalay skgada...@gmail.com wrote: How to redirect the storing of the following files from /tmp to some other location. hadoop-os user-namenode.pid hadoop-os user-datanode.pid yarn-os user-resourcemanager.pid yarn-os user-nodemanager.pid In /tmp, these files are cleared by OS sometime back and I am unable to shutdown by standard scripts stop-dfs.sh or stop-yarn.sh In core-site.xml I have given property hadoop.tmp.dir but still these pid files are present in /tmp only,
Re: No space when running a hadoop job
Thank Aitor. That is what is my observation too. I added a new disk location and manually moved some files. But if 2 locations are given at the beginning itself for dfs.datanode.data.dir, will hadoop balance the disks usage, if not perfect because file sizes may differ. On 9/29/14, Aitor Cedres aced...@pivotal.io wrote: Hi Susheel, Adding a new directory to “dfs.datanode.data.dir” will not balance your disks straightforward. Eventually, by HDFS activity (deleting/invalidating some block, writing new ones), the disks will become balanced. If you want to balance them right after adding the new disk and changing the “dfs.datanode.data.dir” value, you have to shutdown the DN and manually move (mv) some files in the old directory to the new one. The balancer will try to balance the usage between HDFS nodes, but it won't care about internal node disks utilization. For your particular case, the balancer won't fix your issue. Hope it helps, Aitor On 29 September 2014 05:53, Susheel Kumar Gadalay skgada...@gmail.com wrote: You mean if multiple directory locations are given, Hadoop will balance the distribution of files across these different directories. But normally we start with 1 directory location and once it is reaching the maximum, we add new directory. In this case how can we balance the distribution of files? One way is to list the files and move. Will start balance script will work? On 9/27/14, Alexander Pivovarov apivova...@gmail.com wrote: It can read/write in parallel to all drives. More hdd more io speed. On Sep 27, 2014 7:28 AM, Susheel Kumar Gadalay skgada...@gmail.com wrote: Correct me if I am wrong. Adding multiple directories will not balance the files distributions across these locations. Hadoop will add exhaust the first directory and then start using the next, next .. How can I tell Hadoop to evenly balance across these directories. On 9/26/14, Matt Narrell matt.narr...@gmail.com wrote: You can add a comma separated list of paths to the “dfs.datanode.data.dir” property in your hdfs-site.xml mn On Sep 26, 2014, at 8:37 AM, Abdul Navaz navaz@gmail.com wrote: Hi I am facing some space issue when I saving file into HDFS and/or running map reduce job. root@nn:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda2 5.9G 5.9G 0 100% / udev 98M 4.0K 98M 1% /dev tmpfs 48M 192K 48M 1% /run none 5.0M 0 5.0M 0% /run/lock none 120M 0 120M 0% /run/shm overflow 1.0M 4.0K 1020K 1% /tmp /dev/xvda4 7.9G 147M 7.4G 2% /mnt 172.17.253.254:/q/groups/ch-geni-net/Hadoop-NET 198G 108G 75G 59% /groups/ch-geni-net/Hadoop-NET 172.17.253.254:/q/proj/ch-geni-net 198G 108G 75G 59% /proj/ch-geni-net root@nn:~# I can see there is no space left on /dev/xvda2. How can I make hadoop to see newly mounted /dev/xvda4 ? Or do I need to move the file manually from /dev/xvda2 to xvda4 ? Thanks Regards, Abdul Navaz Research Assistant University of Houston Main Campus, Houston TX Ph: 281-685-0388
Extremely amount of memory and DB connections by MR Job
Hi,
I am using a hadoop map reduce job + mongoDb.
It goes against a data base 252Gb big. During the job the amount of conexions
is over 8000 and we gave already 9Gb RAM. The job is still crashing because of
a OutOfMemory with only a 8% of the mapping done.
Are this numbers normal? Or did we miss something regarding configuration?
I attach my code, just in case the problem is with it.
Mapper:
public class AveragePriceMapper extends MapperObject, BasicDBObject, Text,
BSONWritable {
@Override
public void map(final Object key, final BasicDBObject val, final Context
context) throws IOException, InterruptedException {
String id = ;
for(String propertyId : currentId.split(AveragePriceGlobal.SEPARATOR)){
id += val.get(propertyId) + AveragePriceGlobal.SEPARATOR;
}
BSONWritable bsonWritable = new BSONWritable(val);
context.write(new Text(id), bsonWritable);
}
}
Reducer:
public class AveragePriceReducer extends ReducerText, BSONWritable, Text,
Text {
public void reduce(final Text pKey, final IterableBSONWritable pValues,
final Context pContext) throws IOException, InterruptedException {
while(pValues.iterator().hasNext() continueLoop){
BSONWritable next = pValues.iterator().next();
//Make some calculations
}pContext.write(new Text(currentId), new Text(new
MyClass(currentId, AveragePriceGlobal.COMMENT, 0, 0).toString()));
}
}
The configuration includes a query which filters the number of objects to
analyze (not the 252Gb will be analyzed).
Many thanks. Best regards,
Blanca
Re: No space when running a hadoop job
I think they way it works when HDFS has a list in dfs.datanode.data.dir, it's basically a round robin between disks. And yes, it may not be perfect balanced cause of different file sizes. On 29 September 2014 13:15, Susheel Kumar Gadalay skgada...@gmail.com wrote: Thank Aitor. That is what is my observation too. I added a new disk location and manually moved some files. But if 2 locations are given at the beginning itself for dfs.datanode.data.dir, will hadoop balance the disks usage, if not perfect because file sizes may differ. On 9/29/14, Aitor Cedres aced...@pivotal.io wrote: Hi Susheel, Adding a new directory to “dfs.datanode.data.dir” will not balance your disks straightforward. Eventually, by HDFS activity (deleting/invalidating some block, writing new ones), the disks will become balanced. If you want to balance them right after adding the new disk and changing the “dfs.datanode.data.dir” value, you have to shutdown the DN and manually move (mv) some files in the old directory to the new one. The balancer will try to balance the usage between HDFS nodes, but it won't care about internal node disks utilization. For your particular case, the balancer won't fix your issue. Hope it helps, Aitor On 29 September 2014 05:53, Susheel Kumar Gadalay skgada...@gmail.com wrote: You mean if multiple directory locations are given, Hadoop will balance the distribution of files across these different directories. But normally we start with 1 directory location and once it is reaching the maximum, we add new directory. In this case how can we balance the distribution of files? One way is to list the files and move. Will start balance script will work? On 9/27/14, Alexander Pivovarov apivova...@gmail.com wrote: It can read/write in parallel to all drives. More hdd more io speed. On Sep 27, 2014 7:28 AM, Susheel Kumar Gadalay skgada...@gmail.com wrote: Correct me if I am wrong. Adding multiple directories will not balance the files distributions across these locations. Hadoop will add exhaust the first directory and then start using the next, next .. How can I tell Hadoop to evenly balance across these directories. On 9/26/14, Matt Narrell matt.narr...@gmail.com wrote: You can add a comma separated list of paths to the “dfs.datanode.data.dir” property in your hdfs-site.xml mn On Sep 26, 2014, at 8:37 AM, Abdul Navaz navaz@gmail.com wrote: Hi I am facing some space issue when I saving file into HDFS and/or running map reduce job. root@nn:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda2 5.9G 5.9G 0 100% / udev 98M 4.0K 98M 1% /dev tmpfs 48M 192K 48M 1% /run none 5.0M 0 5.0M 0% /run/lock none 120M 0 120M 0% /run/shm overflow 1.0M 4.0K 1020K 1% /tmp /dev/xvda4 7.9G 147M 7.4G 2% /mnt 172.17.253.254:/q/groups/ch-geni-net/Hadoop-NET 198G 108G 75G 59% /groups/ch-geni-net/Hadoop-NET 172.17.253.254:/q/proj/ch-geni-net 198G 108G 75G 59% /proj/ch-geni-net root@nn:~# I can see there is no space left on /dev/xvda2. How can I make hadoop to see newly mounted /dev/xvda4 ? Or do I need to move the file manually from /dev/xvda2 to xvda4 ? Thanks Regards, Abdul Navaz Research Assistant University of Houston Main Campus, Houston TX Ph: 281-685-0388
Re: Hadoop shutdown scripts failing
Check the file $HADOOP_HOME/bin/yarn-daemon.sh; there is a reference to YARN_PID_DIR. If it's not set. it will default to /tmp. On 29 September 2014 13:11, Susheel Kumar Gadalay skgada...@gmail.com wrote: Thanks Aitor. There is a shell variable HADOOP_PID_DIR refererred in hadoop-env.sh but not in yarn-env.sh. On 9/29/14, Aitor Cedres aced...@pivotal.io wrote: Hi Susheel, You have to set in your hadoop-env.sh and yarn-env.sh the variabkes: - HADOOP_PID_DIR - YARN_PID_DIR To point to some other directory (most common is /var/run/hadoop-hdfs hadoop-yarn) Hope it helps, Aitor On 29 September 2014 07:50, Susheel Kumar Gadalay skgada...@gmail.com wrote: How to redirect the storing of the following files from /tmp to some other location. hadoop-os user-namenode.pid hadoop-os user-datanode.pid yarn-os user-resourcemanager.pid yarn-os user-nodemanager.pid In /tmp, these files are cleared by OS sometime back and I am unable to shutdown by standard scripts stop-dfs.sh or stop-yarn.sh In core-site.xml I have given property hadoop.tmp.dir but still these pid files are present in /tmp only,
RE: Extremely amount of memory and DB connections by MR Job
I don't have any experience with MongoDB, but just gave my 2 cents here.
Your code is not efficient, as using the += on String, and you could have
reused the Text object in your mapper, as it is a mutable class, to be reused
and avoid creating it again and again like new Text() in the mapper. My guess
that BSONWritable should be a similar mutable class, if it aims to be used like
the rest Writable Hadoop class.
But even like that, it should just make your mapper run slower, as a lot of
objects need to be GC, instead of OOM.
When you claim 96G ram, I am not sure what do you mean? From what you said, it
failed in mapper stage, so let's focus on mapper. What max heap size you gave
to the mapper task? I don't think 96G is the setting you mean to give to each
mapper task. Otherwise, the only place I can think is that there are millions
of Strings to be appended in one record by += and cause the OOM.
You need to answer the following questions by yourself:
1) Are there any mappers successful?2) The OOM mapper, is it always on the same
block? If so, you need to dig into the source data for that block, to think why
it will cause OOM.3) Did you give reasonable heap size for the mapper? What it
is?
Yong
From: blanca.hernan...@willhaben.at
To: user@hadoop.apache.org
Subject: Extremely amount of memory and DB connections by MR Job
Date: Mon, 29 Sep 2014 12:57:41 +
Hi,
I am using a hadoop map reduce job + mongoDb.
It goes against a data base 252Gb big. During the job the amount of conexions
is over 8000 and we gave already 9Gb RAM. The job is still crashing because of
a OutOfMemory with only a 8% of the mapping done.
Are this numbers normal? Or did we miss something regarding configuration?
I attach my code, just in case the problem is with it.
Mapper:
public class AveragePriceMapper extends MapperObject, BasicDBObject, Text,
BSONWritable {
@Override
public void map(final Object key, final BasicDBObject val, final Context
context) throws IOException, InterruptedException {
String id = ;
for(String propertyId : currentId.split(AveragePriceGlobal.SEPARATOR)){
id += val.get(propertyId) + AveragePriceGlobal.SEPARATOR;
}
BSONWritable bsonWritable = new BSONWritable(val);
context.write(new Text(id), bsonWritable);
}
}
Reducer:
public class AveragePriceReducer extends ReducerText, BSONWritable, Text,
Text {
public void reduce(final Text pKey, final IterableBSONWritable pValues,
final Context pContext) throws IOException, InterruptedException {
while(pValues.iterator().hasNext() continueLoop){
BSONWritable next = pValues.iterator().next();
//Make some calculations
}pContext.write(new Text(currentId), new Text(new
MyClass(currentId, AveragePriceGlobal.COMMENT, 0, 0).toString()));
}
}
The configuration includes a query which filters the number of objects to
analyze (not the 252Gb will be analyzed).
Many thanks. Best regards,
Blanca
AW: Extremely amount of memory and DB connections by MR Job
Thanks for your answer.
To your questions:
1. When you claim 96G ram, I am not sure what do you mean?
It is not 96 Gb RAM, it is 9 Gb that our test server has available (is it too
small?).
2. Your code is not efficient, as using the += on String
I need (or at least I don´t have a better idea) the concatenation of strings
for the emited ID, since I want to group my objects by, e.g. Audi_A3_2010,
another group Audi_A3_2011 And so on. These values are fields in the
objects I get from the DB (BasicDBObject is a MongoDB class).
3. could have reused the Text object in your mapper
I am not sure if I understand your point. I create a new BSONWritable
bsonWritable = new BSONWritable(val); out of my data base object, since the one
given by MongoDB is not mutable, hence not accepted by haddop api as an outpu.
Now your other questions:
1) Are there any mappers successful?
Yes, but after a while, the job seems to need more memory, it runs very slow
until it crashes.
2) The OOM mapper, is it always on the same block? If so, you need to dig into
the source data for that block, to think why it will cause OOM.
I am not sure about this. Is there a hint in the logs to figure it out?
3) Did you give reasonable heap size for the mapper? What it is?
9 Gb (too small??)
Best regards,
Blanca
Von: java8964 [mailto:java8...@hotmail.com]
Gesendet: Montag, 29. September 2014 15:43
An: user@hadoop.apache.org
Betreff: RE: Extremely amount of memory and DB connections by MR Job
I don't have any experience with MongoDB, but just gave my 2 cents here.
Your code is not efficient, as using the += on String, and you could have
reused the Text object in your mapper, as it is a mutable class, to be reused
and avoid creating it again and again like new Text() in the mapper. My guess
that BSONWritable should be a similar mutable class, if it aims to be used like
the rest Writable Hadoop class.
But even like that, it should just make your mapper run slower, as a lot of
objects need to be GC, instead of OOM.
When you claim 96G ram, I am not sure what do you mean? From what you said, it
failed in mapper stage, so let's focus on mapper. What max heap size you gave
to the mapper task? I don't think 96G is the setting you mean to give to each
mapper task. Otherwise, the only place I can think is that there are millions
of Strings to be appended in one record by += and cause the OOM.
You need to answer the following questions by yourself:
1) Are there any mappers successful?
2) The OOM mapper, is it always on the same block? If so, you need to dig into
the source data for that block, to think why it will cause OOM.
3) Did you give reasonable heap size for the mapper? What it is?
Yong
From: blanca.hernan...@willhaben.atmailto:blanca.hernan...@willhaben.at
To: user@hadoop.apache.orgmailto:user@hadoop.apache.org
Subject: Extremely amount of memory and DB connections by MR Job
Date: Mon, 29 Sep 2014 12:57:41 +
Hi,
I am using a hadoop map reduce job + mongoDb.
It goes against a data base 252Gb big. During the job the amount of conexions
is over 8000 and we gave already 9Gb RAM. The job is still crashing because of
a OutOfMemory with only a 8% of the mapping done.
Are this numbers normal? Or did we miss something regarding configuration?
I attach my code, just in case the problem is with it.
Mapper:
public class AveragePriceMapper extends MapperObject, BasicDBObject, Text,
BSONWritable {
@Override
public void map(final Object key, final BasicDBObject val, final Context
context) throws IOException, InterruptedException {
String id = ;
for(String propertyId : currentId.split(AveragePriceGlobal.SEPARATOR)){
id += val.get(propertyId) + AveragePriceGlobal.SEPARATOR;
}
BSONWritable bsonWritable = new BSONWritable(val);
context.write(new Text(id), bsonWritable);
}
}
Reducer:
public class AveragePriceReducer extends ReducerText, BSONWritable, Text,
Text {
public void reduce(final Text pKey, final IterableBSONWritable pValues,
final Context pContext) throws IOException, InterruptedException {
while(pValues.iterator().hasNext() continueLoop){
BSONWritable next = pValues.iterator().next();
//Make some calculations
}pContext.write(new Text(currentId), new Text(new
MyClass(currentId, AveragePriceGlobal.COMMENT, 0, 0).toString()));
}
}
The configuration includes a query which filters the number of objects to
analyze (not the 252Gb will be analyzed).
Many thanks. Best regards,
Blanca
RE: Re: Regarding HDFS and YARN support for S3
Hi Takenori, Thanks for replying but still seem not getting some concepts I understand that we need to give fs.AbstractFileSystem.s3.impl if we want to submit job using ./yarn jar with S3 HCFS configured. But what i don't understand is why 2 interfaces (may be i am novice in HDFS and hence not able to completely correlate with jira's which you gave). If you can brief the differences between FileSystem and AbstractFileSystem, It would be helpful. Regards, Naga Huawei Technologies Co., Ltd. Phone: Fax: Mobile: +91 9980040283 Email: naganarasimh...@huawei.commailto:naganarasimh...@huawei.com Huawei Technologies Co., Ltd. Bantian, Longgang District,Shenzhen 518129, P.R.China http://www.huawei.com From: Takenori Sato [ts...@cloudian.com] Sent: Monday, September 29, 2014 07:29 To: user@hadoop.apache.org Subject: Re: Re: Regarding HDFS and YARN support for S3 Hi, You may want to check HADOOP-10400https://issues.apache.org/jira/browse/HADOOP-10400 for the overhaul of S3 filesystem fixed in 2.6. The subclass of AbstractFileSystem was filed as HADOOP-10643https://issues.apache.org/jira/browse/HADOOP-10643, but which was not included in HADOOP-10400 though I made a commenthttps://issues.apache.org/jira/browse/HADOOP-10400?focusedCommentId=14104967page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14104967. I suggest not to use S3 as defaultFS as commented in Why you cannot use S3 as a replacement for HDFShttps://wiki.apache.org/hadoop/AmazonS3 to avoid all sorts of these issues. The best practice is to use S3 as a supplementary solution to Hadoop in order to bring life cycle management(expiration and tiering), and source/destination over the internet. Thanks, Takenori On Sun, Sep 28, 2014 at 5:23 PM, Naganarasimha G R (Naga) garlanaganarasi...@huawei.commailto:garlanaganarasi...@huawei.com wrote: Hi Jay, Thanks a lot for replying and it clarifies most of it, but still some parts are not so clear . Some clarifications from my side : | When you say HDFS does not support fs.AbstractFileSystem.s3.impl That is true. If your file system is configured using HDFS, then s3 urls will not be used, ever. :) i think i am not doing this basic mistake . What we have done is we have configured viewfs://nsX for fs.defaultFS and one of the mount is S3 i.e. fs.viewfs.mounttable.nsX.link./uds to s3://hadoop/test1/. So it fails to even create YARNRunner instance as there is no mapping for fs.AbstractFileSystem.s3.impl if run ./yarn jar. But as per the code even if set fs.defaultFS to s3 it will not work as there is no mapping for S3's impl of AbstractFileSystem interface. These are my further queries 1. Whats the purpose of AbstractFileSystem and FileSystem interfaces? 2. Does HDFS default package(code) support configuration of S3 ? I see S3 implementation of FileSystem interface(org.apache.hadoop.fs.s3.S3FileSystem) but not for AbstractFileSystem !. So i presume it doesn't support S3 completely. Whats the reason for not supporting both ? 3. Suppose if i need to support Amazon S3 do i need to extend and implement AbstractFileSystem and configure fs.AbstractFileSystem.s3.impl or some thing more than this i need to take care? Regards, Naga Huawei Technologies Co., Ltd. Phone: Fax: Mobile: +91 9980040283tel:%2B91%209980040283 Email: naganarasimh...@huawei.commailto:naganarasimh...@huawei.com Huawei Technologies Co., Ltd. Bantian, Longgang District,Shenzhen 518129, P.R.China http://www.huawei.com From: jay vyas [jayunit100.apa...@gmail.commailto:jayunit100.apa...@gmail.com] Sent: Saturday, September 27, 2014 02:41 To: common-u...@hadoop.apache.orgmailto:common-u...@hadoop.apache.org Subject: Re: See https://wiki.apache.org/hadoop/HCFS/ YES Yarn is written to the FileSystem interface. It works on S3FileSystem and GlusterFileSystem and any other HCFS. We have run , and continue to run, the many tests in apache bigtop's test suite against our hadoop clusters running on alternative file system implementations, and it works. When you say HDFS does not support fs.AbstractFileSystem.s3.impl That is true. If your file system is configured using HDFS, then s3 urls will not be used, ever. When you create a FileSystem object in hadoop, it reads the uri (i.e. glusterfs:///) and then finds the file system binding in your core-site.xml (i.e. fs.AbstractFileSystem.glusterfs.impl). So the URI must have a corresponding entry in the core-site.xml. As a reference implementation, you can see https://github.com/gluster/glusterfs-hadoop/blob/master/conf/core-site.xml On Fri, Sep 26, 2014 at 10:10 AM, Naganarasimha G R (Naga) garlanaganarasi...@huawei.commailto:garlanaganarasi...@huawei.com wrote: Hi All, I have following doubts on pluggable FileSystem and YARN 1. If all the implementations should extend FileSystem then why there is a parallel class AbstractFileSystem. which ViewFS extends ? 2. Is YARN supposed
Re: Re: Regarding HDFS and YARN support for S3
Hi Naga, But what i don't understand is why 2 interfaces (may be i am novice in HDFS and hence not able to completely correlate with jira's which you gave). A client program is encouraged to use FileContext API instead of FileSystem API. Here's why http://www.slideshare.net/hadoopusergroup/file-context. And the whole discussion is at HADOOP-6223(New improved FileSystem interface for those implementing new files systems.). Thanks, Takenori On Mon, Sep 29, 2014 at 11:27 PM, Naganarasimha G R (Naga) garlanaganarasi...@huawei.com wrote: Hi Takenori, Thanks for replying but still seem not getting some concepts I understand that we need to give ***fs.AbstractFileSystem.s3.impl *if we want to submit job using ./yarn jar with S3 HCFS configured*. * But what i don't understand is why 2 interfaces (may be i am novice in HDFS and hence not able to completely correlate with jira's which you gave). If you can brief the differences between FileSystem and AbstractFileSystem, It would be helpful. Regards, Naga Huawei Technologies Co., Ltd. Phone: Fax: Mobile: +91 9980040283 Email: naganarasimh...@huawei.com Huawei Technologies Co., Ltd. Bantian, Longgang District,Shenzhen 518129, P.R.China http://www.huawei.com *From:* Takenori Sato [ts...@cloudian.com] *Sent:* Monday, September 29, 2014 07:29 *To:* user@hadoop.apache.org *Subject:* Re: Re: Regarding HDFS and YARN support for S3 Hi, You may want to check HADOOP-10400 https://issues.apache.org/jira/browse/HADOOP-10400 for the overhaul of S3 filesystem fixed in 2.6. The subclass of AbstractFileSystem was filed as HADOOP-10643 https://issues.apache.org/jira/browse/HADOOP-10643, but which was not included in HADOOP-10400 though I made a comment https://issues.apache.org/jira/browse/HADOOP-10400?focusedCommentId=14104967page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14104967 . I suggest not to use S3 as defaultFS as commented in Why you cannot use S3 as a replacement for HDFS https://wiki.apache.org/hadoop/AmazonS3 to avoid all sorts of these issues. The best practice is to use S3 as a supplementary solution to Hadoop in order to bring life cycle management(expiration and tiering), and source/destination over the internet. Thanks, Takenori On Sun, Sep 28, 2014 at 5:23 PM, Naganarasimha G R (Naga) garlanaganarasi...@huawei.com wrote: Hi Jay, Thanks a lot for replying and it clarifies most of it, but still some parts are not so clear . Some clarifications from my side : *| When you say HDFS does not support fs.AbstractFileSystem.s3.impl That is true. If your file system is configured using HDFS, then s3 urls will not be used, ever.* :) i think i am not doing this basic mistake . What we have done is we have configured *viewfs://nsX for fs.defaultFS* and one of the mount is S3 i.e. *fs.viewfs.mounttable.nsX.link./uds to s3://hadoop/test1/* . So it fails to even create YARNRunner instance as there is no mapping for ***fs.AbstractFileSystem.s3.impl *if run ./yarn jar*. *But as per the code even if set *fs.defaultFS* to s3 it will not work as there is no mapping for S3's impl of AbstractFileSystem interface. These are my further queries 1. Whats the purpose of *AbstractFileSystem *and *FileSystem * interfaces? 2. Does HDFS default package(code) support configuration of S3 ? I see S3 implementation of *FileSystem* interface( *org.apache.hadoop.fs.s3.S3FileSystem*) *but not for **AbstractFileSystem **!. *So i presume it doesn't support S3 completely. Whats the reason for not supporting both ? 3. Suppose if i need to support Amazon S3 do i need to extend and implement *AbstractFileSystem *and configure ***fs.AbstractFileSystem.s3.impl *or some thing more than this i need to take care*?* Regards, Naga Huawei Technologies Co., Ltd. Phone: Fax: Mobile: +91 9980040283 Email: naganarasimh...@huawei.com Huawei Technologies Co., Ltd. Bantian, Longgang District,Shenzhen 518129, P.R.China http://www.huawei.com -- *From:* jay vyas [jayunit100.apa...@gmail.com] *Sent:* Saturday, September 27, 2014 02:41 *To:* common-u...@hadoop.apache.org *Subject:* Re: See https://wiki.apache.org/hadoop/HCFS/ YES Yarn is written to the FileSystem interface. It works on S3FileSystem and GlusterFileSystem and any other HCFS. We have run , and continue to run, the many tests in apache bigtop's test suite against our hadoop clusters running on alternative file system implementations, and it works. When you say HDFS does not support fs.AbstractFileSystem.s3.impl That is true. If your file system is configured using HDFS, then s3 urls will not be used, ever. When you create a FileSystem object in hadoop, it reads the uri (i.e. glusterfs:///) and then finds the file system binding in your core-site.xml (i.e. fs.AbstractFileSystem.glusterfs.impl). So
RE: AW: Extremely amount of memory and DB connections by MR Job
Here are my suggestions originally aims to improve the efficient:
1) In your case, you could use StringBuilder, which has the append method,
should be more efficient to concatenate your string data in this case.2) What I
mean to reuse the Text object is as following: public class mapper extends
Mapper () { private Text data = new Text(); @Override
public void map(final Object key, final BasicDBObject val, final Context
context) throws IOException, InterruptedException {//
instead of do new Text(id) // you can always use the
following way data.set(id);
context.write(data, bsonWritable);}As you can see, you avoid to
create lots, lots of Text object in the map method. This method could be
invoked a lot of times. In this way, you avoid asking GC to clean a lot of Text
object, by reusing the same Text object per map. I believe you can do the same
for BSONWritable. Check the javadoc for that class.3) 9G is a lot of heap for a
map task. How many map tasks your job generates? Are your source splitable? For
one block data (I assume it is 128M or 256M), I cannot image you need 9G heap
for mapper. Your OOM maybe caused by that your job runs out of physical memory
of all the concurrent running mapper tasks.
1) How many total mapper tasks being generated in your job?2) How many
data/task nodes you have in your cluster? On the OOM node, how many mapper
tasks being kicked off? You can find all these information in the JobTracker in
MR1, or AM in MR2.3) If each mapper assigned 9G memory, and there are multi
mappers running in the OOM node, how much real physical memory you have? 4) You
can see the input source for each mapper task in JobTracker or AM. If failed
mapper is always for the same block, then research that source data file. You
need to have real good reason to allocate 9G heap for a mapper task. Did you
originally start from 1G?
Yong
From: blanca.hernan...@willhaben.at
To: user@hadoop.apache.org
Subject: AW: Extremely amount of memory and DB connections by MR Job
Date: Mon, 29 Sep 2014 14:16:24 +
Thanks for your answer.
To your questions:
1.
When you claim 96G ram, I am not sure what do you mean?
It is not 96 Gb RAM, it is 9 Gb that our test server has available (is it too
small?).
2.
Your code is not efficient, as using the += on String
I need (or at least I don´t have a better idea) the concatenation of strings
for the emited ID, since I want
to group my objects by, e.g. Audi_A3_2010, another group Audi_A3_2011…. And so
on. These values are fields in the objects I get from the DB (BasicDBObject is
a MongoDB class).
3.
could have reused the Text object in your mapper
I am not sure if I understand your point. I create a new BSONWritable
bsonWritable = new BSONWritable(val); out of my
data base object, since the one given by MongoDB is not mutable, hence not
accepted by haddop api as an outpu.
Now your other questions:
1) Are there any mappers successful?
Yes, but after a while, the job seems to need more memory, it runs very slow
until it crashes.
2) The OOM mapper, is it always on the same block? If so, you need to dig into
the source data for that block, to think why it will cause OOM.
I am not sure about this. Is there a hint in the logs to figure it out?
3) Did you give reasonable heap size for the mapper? What it is?
9 Gb (too small??)
Best regards,
Blanca
Von: java8964 [mailto:java8...@hotmail.com]
Gesendet: Montag, 29. September 2014 15:43
An: user@hadoop.apache.org
Betreff: RE: Extremely amount of memory and DB connections by MR Job
I don't have any experience with MongoDB, but just gave my 2 cents here.
Your code is not efficient, as using the += on String, and you could have
reused the Text object in your mapper, as it is a mutable class, to be reused
and avoid creating it again and again
like new Text() in the mapper. My guess that BSONWritable should be a
similar mutable class, if it aims to be used like the rest Writable Hadoop
class.
But even like that, it should just make your mapper run slower, as a lot of
objects need to be GC, instead of OOM.
When you claim 96G ram, I am not sure what do you mean? From what you said, it
failed in mapper stage, so let's focus on mapper. What max heap size you gave
to the mapper task? I don't think
96G is the setting you mean to give to each mapper task. Otherwise, the only
place I can think is that there are millions of Strings to be appended in one
record by += and cause the OOM.
You need to answer the following questions by yourself:
1) Are there any mappers successful?
2) The OOM mapper, is it always on the same block? If so, you need to dig into
the source data for that block, to think why it will cause OOM.
3) Did you give reasonable heap size for the mapper? What it is?
Yong
Using Yarn in end to end tests
I am currently developing tests that use a mini yarn cluster. Because it is running on circle-ci I need to use the absolute minimum amount of memory. I'm currently setting conf.setFloat(yarn. nodemanager.vmem-pmem-ratio, 8.0f); conf.setBoolean(mapreduce.map.speculative, false); conf.setBoolean(mapreduce.reduce.speculative, false); conf.setInt(yarn.scheduler.minimum-allocation-mb, 128); conf.setInt(yarn.scheduler.maximum-allocation-mb, 256); conf.setInt(yarn.nodemanager.resource.memory-mb, 256); conf.setInt(mapreduce.map.memory.mb, 128); conf.set(mapreduce.map.java.opts, -Xmx128m); conf.setInt(mapreduce.reduce.memory.mb, 128); conf.set(mapreduce.reduce.java.opts, -Xmx128m); conf.setInt(mapreduce.task.io.sort.mb, 64); conf.setInt(yarn.app.mapreduce.am.resource.mb, 128); conf.set(yarn.app.mapreduce.am.command-opts, -Xmx109m); conf.setInt(yarn.scheduler.minimum-allocation-vcores, 1); conf.setInt(yarn.scheduler.maximum-allocation-vcores, 1); conf.setInt(yarn.nodemanager.resource.cpu-vcores, 1); conf.setInt(mapreduce.map.cpu.vcore, 1); conf.setInt(mapreduce.reduce.cpu.vcore, 1); conf.setInt(mapreduce.tasktracker.map.tasks.maximum, 1); conf.setInt(mapreduce.tasktracker.reduce.tasks.maximum, 1); conf.setInt(yarn.scheduler.capacity.root.capacity,1); conf.setInt(yarn.scheduler.capacity.maximum-applications, 1); conf.setInt(mapreduce.jobtracker.taskscheduler.maxrunningtasks.perjob, 1); but I am still seeing many child tasks running https://circle-artifacts.com/gh/OhmData/hbase-public/314/artifacts/2/tmp/memory-usage.txt Any ideas on how to actually limit yarn to one or two children at a time?
Re: Hadoop shutdown scripts failing
Thanks On 9/29/14, Aitor Cedres aced...@pivotal.io wrote: Check the file $HADOOP_HOME/bin/yarn-daemon.sh; there is a reference to YARN_PID_DIR. If it's not set. it will default to /tmp. On 29 September 2014 13:11, Susheel Kumar Gadalay skgada...@gmail.com wrote: Thanks Aitor. There is a shell variable HADOOP_PID_DIR refererred in hadoop-env.sh but not in yarn-env.sh. On 9/29/14, Aitor Cedres aced...@pivotal.io wrote: Hi Susheel, You have to set in your hadoop-env.sh and yarn-env.sh the variabkes: - HADOOP_PID_DIR - YARN_PID_DIR To point to some other directory (most common is /var/run/hadoop-hdfs hadoop-yarn) Hope it helps, Aitor On 29 September 2014 07:50, Susheel Kumar Gadalay skgada...@gmail.com wrote: How to redirect the storing of the following files from /tmp to some other location. hadoop-os user-namenode.pid hadoop-os user-datanode.pid yarn-os user-resourcemanager.pid yarn-os user-nodemanager.pid In /tmp, these files are cleared by OS sometime back and I am unable to shutdown by standard scripts stop-dfs.sh or stop-yarn.sh In core-site.xml I have given property hadoop.tmp.dir but still these pid files are present in /tmp only,
binding namenode and job tracker to 0.0.0.0
Hi , I have 2 different networks in my setup Job tracker and name node are running in private network, eclipse client is running on public network I see the JIRA which is very relevant is there a workaround ? https://issues.apache.org/jira/browse/HADOOP-1202 -- Warm Regards, *Bharath Kumar *
Re: No space when running a hadoop job
Dear All, I am not doing load balancing here. I am just copying a file and it is throwing me an error no space left on the device. hduser@dn1:~$ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda2 5.9G 5.1G 533M 91% / udev 98M 4.0K 98M 1% /dev tmpfs 48M 196K 48M 1% /run none 5.0M 0 5.0M 0% /run/lock none 120M 0 120M 0% /run/shm 172.17.253.254:/q/groups/ch-geni-net/Hadoop-NET 198G 116G 67G 64% /groups/ch-geni-net/Hadoop-NET 172.17.253.254:/q/proj/ch-geni-net 198G 116G 67G 64% /proj/ch-geni-net /dev/xvda4 7.9G 147M 7.4G 2% /mnt hduser@dn1:~$ hduser@dn1:~$ hduser@dn1:~$ hduser@dn1:~$ cp data2.txt data3.txt cp: writing `data3.txt': No space left on device cp: failed to extend `data3.txt': No space left on device hduser@dn1:~$ I guess by default it is copying to default location. Why I am getting this error ? How can I fix this ? Thanks Regards, Abdul Navaz Research Assistant University of Houston Main Campus, Houston TX Ph: 281-685-0388 From: Aitor Cedres aced...@pivotal.io Reply-To: user@hadoop.apache.org Date: Monday, September 29, 2014 at 7:53 AM To: user@hadoop.apache.org Subject: Re: No space when running a hadoop job I think they way it works when HDFS has a list in dfs.datanode.data.dir, it's basically a round robin between disks. And yes, it may not be perfect balanced cause of different file sizes. On 29 September 2014 13:15, Susheel Kumar Gadalay skgada...@gmail.com wrote: Thank Aitor. That is what is my observation too. I added a new disk location and manually moved some files. But if 2 locations are given at the beginning itself for dfs.datanode.data.dir, will hadoop balance the disks usage, if not perfect because file sizes may differ. On 9/29/14, Aitor Cedres aced...@pivotal.io wrote: Hi Susheel, Adding a new directory to ³dfs.datanode.data.dir² will not balance your disks straightforward. Eventually, by HDFS activity (deleting/invalidating some block, writing new ones), the disks will become balanced. If you want to balance them right after adding the new disk and changing the ³dfs.datanode.data.dir² value, you have to shutdown the DN and manually move (mv) some files in the old directory to the new one. The balancer will try to balance the usage between HDFS nodes, but it won't care about internal node disks utilization. For your particular case, the balancer won't fix your issue. Hope it helps, Aitor On 29 September 2014 05:53, Susheel Kumar Gadalay skgada...@gmail.com wrote: You mean if multiple directory locations are given, Hadoop will balance the distribution of files across these different directories. But normally we start with 1 directory location and once it is reaching the maximum, we add new directory. In this case how can we balance the distribution of files? One way is to list the files and move. Will start balance script will work? On 9/27/14, Alexander Pivovarov apivova...@gmail.com wrote: It can read/write in parallel to all drives. More hdd more io speed. On Sep 27, 2014 7:28 AM, Susheel Kumar Gadalay skgada...@gmail.com wrote: Correct me if I am wrong. Adding multiple directories will not balance the files distributions across these locations. Hadoop will add exhaust the first directory and then start using the next, next .. How can I tell Hadoop to evenly balance across these directories. On 9/26/14, Matt Narrell matt.narr...@gmail.com wrote: You can add a comma separated list of paths to the ³dfs.datanode.data.dir² property in your hdfs-site.xml mn On Sep 26, 2014, at 8:37 AM, Abdul Navaz navaz@gmail.com wrote: Hi I am facing some space issue when I saving file into HDFS and/or running map reduce job. root@nn:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda2 5.9G 5.9G 0 100% / udev 98M 4.0K 98M 1% /dev tmpfs 48M 192K 48M 1% /run none 5.0M 0 5.0M 0% /run/lock none 120M 0 120M 0% /run/shm overflow 1.0M 4.0K 1020K 1% /tmp /dev/xvda4 7.9G 147M 7.4G 2% /mnt 172.17.253.254:/q/groups/ch-geni-net/Hadoop-NET 198G 108G 75G 59% /groups/ch-geni-net/Hadoop-NET
YARN application failing to localize needed jars (Xpost: hbase users)
Hi folks, I'm having trouble using HBASE copyTable to seed an existing tables data to a replication peer. Surely its an oversight in configuration on our part, but I've scoured the web and doc's for a couple days now. We have been able to run these jobs with success (perhaps they don't require localization of any jars since the jars are already included in the environment on each nodemanager?): ./hadoop org.apache.hadoop.fs.TestDFSIO -write -nrFiles 5 -fileSize 5GB ./hbase org.apache.hadoop.hbase.PerformanceEvaluation randomWrite 5 When running copyTable it seems like the resources are placed on the yarn application manager but not localized to the nodemanagers. I've tried several settings for yarn.app.mapreduce.am.staging-dir and in the below iteration im trying the defaults, the path exists with correct permissions on each NM/AM hadoop.tmp.dir is specified in core-site.xml Here is the full command we are running: [hbase@master2 bin]$ ./hbase org.apache.hadoop.hbase.mapreduce.CopyTable --starttime=1409128964 --peer.adr=master0.hbasex1.test.cloud.domain.tv, master1.hbasex1.test.cloud.v.tv,master2.hbasex1.test.cloud.domain.tv:2181:/hbase moderation SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hbase-0.98.6.1-1a/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.5.0-1a/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 2014-09-26 23:51:24,141 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-09-26 23:51:24,863 WARN [main] shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2014-09-26 23:51:24,910 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.HConstants, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,911 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-protocol-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.client.Put, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-client-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.CompatibilityFactory, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,913 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.zookeeper.ZooKeeper, using jar /usr/local/hbase-0.98.6.1-1a/lib/zookeeper-3.4.6.jar 2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.jboss.netty.channel.ChannelFactory, using jar /usr/local/hbase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar 2014-09-26 23:51:24,915 DEBUG [main] mapreduce.TableMapReduceUtil: For class com.google.protobuf.Message, using jar /usr/local/hbase-0.98.6.1-1a/lib/protobuf-java-2.5.0.jar 2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For class com.google.common.collect.Lists, using jar /usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar 2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.cloudera.htrace.Trace, using jar /usr/local/hbase-0.98.6.1-1a/lib/htrace-core-2.04.jar 2014-09-26 23:51:24,917 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.cliffc.high_scale_lib.Counter, using jar /usr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar 2014-09-26 23:51:24,921 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.LongWritable, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,922 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.Text, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,923 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,923 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.LongWritable, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,924 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.Text, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,925 DEBUG [main] mapreduce.TableMapReduceUtil: For class
Kerberosed Hadoop HDFS
Hi Experts:
I write the following java program to access Kerberosed Hadoop File system:
---
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileStatus;
import java.net.URI;
public class HDFSExample {
public static void main(String[] args) throws Exception{
Path path = new Path(/user/sochen);
if ( args.length == 1){
path = new Path(args[0]);
}
Configuration conf = new Configuration();
System.out.println(fs.defaultFS = + conf.get(fs.defaultFS));
URI uri = new URI(hdfs://my.namenode.com:8020);
FileSystem fs = FileSystem.get(uri, conf);
System.out.println(List of files in + path);
FileStatus [] files = fs.listStatus(path);
for (FileStatus file : files ){
System.out.println(file.getPath().getName());
}
}
}
--
It work fine as long as I issued kinit for my principal sochen
(which is also my current linux login account user name).
But if I changed one line above from: FileSystem fs =
FileSystem.get(uri, conf); to FileSystem fs = FileSystem.get(uri,
conf, sochen), I will hit below error:
14/09/29 13:47:32 ERROR security.UserGroupInformation:
PriviledgedActionException as:sochen (auth:SIMPLE)
cause:javax.security.sasl.SaslException: GSS initiate failed [Caused
by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]
What I want to do is: to specify a username sochen to explicitly
to get a FileSystem, but underneath hadoop changed the authentication
mode to SIMPLE , why ?
Then how can I get a FileSystem based on a login user ?
Thanks.
Sophia
Re: From java application : how to access kerberosed hadoop HDFS ?
Resolved: from client machine I need to update the ./etc/hadoop/core-site.xml with site specific info e.g. property namehadoop.security.authentication/name valuekerberos/value /property also hdfs-site.xml needs: property namedfs.namenode.kerberos.principal/name valuehdfs/_HOST@your-realmname/value /property property namedfs.namenode.kerberos.internal.spnego.principal/name valueHTTP/_HOST@your-realmname/value /property property namedfs.datanode.kerberos.principal/name valuehdfs/_HOST@your-realmname/value /property Thanks. Sophia On Sat, Sep 27, 2014 at 8:13 PM, Liu, Yi A yi.a@intel.com wrote: You should configure hadoop.security.authentication to Kerberos in your core-site.xml. Please refer to http://hadoop.apache.org/docs/r2.5.1/hadoop-project-dist/hadoop-common/SecureMode.html Regards, Yi Liu -Original Message- From: Xiaohua Chen [mailto:xiaohua.c...@gmail.com] Sent: Saturday, September 27, 2014 11:51 AM To: user@hadoop.apache.org Subject: From java application : how to access kerberosed hadoop HDFS ? Hi , We recently has added kerberos on our CDH4 cluster, and our java application hit error: Caused by: org.apache.hadoop.security.AccessControlException: Authorization (hadoop.security.authorization) is enabled but authentication (hadoop.security.authentication) is configured as simple. Please configure another method like kerberos or digest. The above error is caused by below code: mFileSystem = FileSystem.get(uri,new Configuration(), loginUser); FileStatus[] statusArray = mFileSystem.listStatus(pathName) I am new to this area and can you shed some light on: how to configure the authentication method to Kerberos to avoid the above error ? Thanks and regards, Sophia
how to access oozie web console for kerberosed hadoop
Hi Experts: I can go to oozie console http://oozieserver:11000/oozie/ when our hadoop cluster has no kerberos setup. But after Keberos setup, I try the same oozie console from IE, i got error: type Status report message org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag) description This request requires HTTP authentication (org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)). Can you let me know what configuration steps I need to access the oozie console ? Thanks. Sophia
Re: Hadoop UI - Unable to connect to the application master from the Hadoop UI.
The host name is fully qualified , meaning there is nothing more that I can add , it just seems the ports might be messed up , but I don't know which ones On Mon, Sep 29, 2014 at 12:44 AM, Susheel Kumar Gadalay skgada...@gmail.com wrote: I also faced some issue like this. It shows the URL in host name:port Copy paste the link in browser and expand the host name. I set up the host names in windows user etc/hosts file but still it could not resolve. On 9/29/14, S.L simpleliving...@gmail.com wrote: Hi All, I am running a 3 node Apache Hadoop YARN 2.3.0 cluster , after a job is submitted , when I ascess the application master from the UI I get the following exception and I am unable to connect to the Application Master from the UI, can someone let me know , what I need to look at? HTTP ERROR 500 Problem accessing /proxy/application_1411841629814_0032/. Reason: Connection refused Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at java.net.Socket.init(Socket.java:425) at java.net.Socket.init(Socket.java:280) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:185) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:336) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1183) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at
Re: Failed to active namenode when config HA
Hi, Matt Thank you very much for your response! There were some mistakes in my description as i wrote this mail in a hurry. I put those properties is in hdfs-site.xml not core-site.xml. There are four name nodes because i also using HDFS federation, so there are two nameservices in porperty namedfs.nameservices/name and each nameservice will have two namenodes. If i configure only HA (only one nameservice), everything is ok, and HAAdmin can determine the namenodes nn1, nn3. But if i configure two nameservice and set namenodes nn1,nn3 for nameservice1 and nn2,nn4 for nameservices2. I can start these namenodes successfully and the namenodes are all in standby state at th beginning. But if i want to change one namenode to active state, use command hdfs haadmin -transitionToActive nn1 HAAdmin throw exception as it cannot determine the four namenodes(nn1,nn2,nn3,nn4) at all. Do you used to configure HAFederation and know what may cause these problem? Thanks, Lucy -- Original -- From: Matt Narrell;matt.narr...@gmail.com; Send time: Monday, Sep 29, 2014 6:28 AM To: useruser@hadoop.apache.org; Subject: Re: Failed to active namenode when config HA I??m pretty sure HDFS HA is relegated to two name nodes (not four), designated active and standby. Secondly, I believe these properties should be in hdfs-site.xml NOT core-site.xml. Furthermore, I think your HDFS nameservices are misconfigured. Consider the following: ?xml version=1.0? configuration property namedfs.replication/name value3/value /property property namedfs.namenode.name.dir/name valuefile:/var/data/hadoop/hdfs/nn/value /property property namedfs.datanode.data.dir/name valuefile:/var/data/hadoop/hdfs/dn/value /property property namedfs.ha.automatic-failover.enabled/name valuetrue/value /property property namedfs.nameservices/name valuehdfs-cluster/value /property property namedfs.ha.namenodes.hdfs-cluster/name valuenn1,nn2/value /property property namedfs.namenode.rpc-address.hdfs-cluster.nn1/name valuenamenode1:8020/value /property property namedfs.namenode.http-address.hdfs-cluster.nn1/name valuenamenode1:50070/value /property property namedfs.namenode.rpc-address.hdfs-cluster.nn2/name valuenamenode2:8020/value /property property namedfs.namenode.http-address.hdfs-cluster.nn2/name valuenamenode2:50070/value /property property namedfs.namenode.shared.edits.dir/name valueqjournal://journalnode1:8485;journalnode2:8485;journalnode3:8485/hdfs-cluster/value /property property namedfs.client.failover.proxy.provider.hdfs-cluster/name valueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value /property property namedfs.ha.fencing.methods/name valuesshfence/value /property property namedfs.ha.fencing.ssh.private-key-files/name value/home/hadoop/.ssh/id_rsa/value /property /configuration mn On Sep 28, 2014, at 12:56 PM, ?? 475053...@qq.com wrote: Hi, I'm new to hadoop and meet some problems when config HA. Below are some important configuration in core-site.xml property namedfs.nameservices/name valuens1,ns2/value /property property namedfs.ha.namenodes.ns1/name valuenn1,nn3/value /property property namedfs.ha.namenodes.ns2/name valuenn2,nn4/value /property property namedfs.namenode.rpc-address.ns1.nn1/name valuenamenode1:9000/value /property property namedfs.namenode.rpc-address.ns1.nn3/name valuenamenode3:9000/value /property property namedfs.namenode.rpc-address.ns2.nn2/name valuenamenode2:9000/value /property property namedfs.namenode.rpc-address.ns2.nn4/name valuenamenode4:9000/value /property property namedfs.namenode.shared.edits.dir/name valueqjournal://datanode2:8485;datanode3:8485;datanode4:8485/ns1/value /property property namedfs.client.failover.proxy.provider.ns1/name valueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value /property property namedfs.ha.fencing.methods/name valuesshfence/value /property property namedfs.ha.fencing.ssh.private-key-files/name value/home/hduser/.ssh/id_rsa/value /property property namedfs.ha.fencing.ssh.connect-timeout/name value3/value /property property namedfs.journalnode.edits.dir/name value/home/hduser/mydata/hdfs/journalnode/value /property (two nameservice ns1,ns2 is for configuring federation later. In this step, I only want launch ns1 on namenode1,namenode3) After configuration, I did the following steps firstly, I start jornalnode on datanode2,datanode3,datanode4 secondly I
RE: Failed to active namenode when config HA
You need to start the ZKFC process which will monitor and manage the state of namenode. Automatic failover adds two new components to an HDFS deployment: a ZooKeeper quorum, and the ZKFailoverController process (abbreviated as ZKFC). Apache ZooKeeper is a highly available service for maintaining small amounts of coordination data, notifying clients of changes in that data, and monitoring clients for failures. The implementation of automatic HDFS failover relies on ZooKeeper for the following things: * Failure detection - each of the NameNode machines in the cluster maintains a persistent session in ZooKeeper. If the machine crashes, the ZooKeeper session will expire, notifying the other NameNode that a failover should be triggered. * Active NameNode election - ZooKeeper provides a simple mechanism to exclusively elect a node as active. If the current active NameNode crashes, another node may take a special exclusive lock in ZooKeeper indicating that it should become the next active. The ZKFailoverController (ZKFC) is a new component which is a ZooKeeper client which also monitors and manages the state of the NameNode. Each of the machines which runs a NameNode also runs a ZKFC, and that ZKFC is responsible for: * Health monitoring - the ZKFC pings its local NameNode on a periodic basis with a health-check command. So long as the NameNode responds in a timely fashion with a healthy status, the ZKFC considers the node healthy. If the node has crashed, frozen, or otherwise entered an unhealthy state, the health monitor will mark it as unhealthy. * ZooKeeper session management - when the local NameNode is healthy, the ZKFC holds a session open in ZooKeeper. If the local NameNode is active, it also holds a special lock znode. This lock uses ZooKeeper's support for ephemeral nodes; if the session expires, the lock node will be automatically deleted. * ZooKeeper-based election - if the local NameNode is healthy, and the ZKFC sees that no other node currently holds the lock znode, it will itself try to acquire the lock. If it succeeds, then it has won the election, and is responsible for running a failover to make its local NameNode active. The failover process is similar to the manual failover described above: first, the previous active is fenced if necessary, and then the local NameNode transitions to active state. Please go through following link for more details.. http://hadoop.apache.org/docs/r2.5.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html Thanks Regards Brahma Reddy Battula From: 清如许 [475053...@qq.com] Sent: Tuesday, September 30, 2014 8:54 AM To: user Subject: Re: Failed to active namenode when config HA Hi, Matt Thank you very much for your response! There were some mistakes in my description as i wrote this mail in a hurry. I put those properties is in hdfs-site.xml not core-site.xml. There are four name nodes because i also using HDFS federation, so there are two nameservices in porperty namedfs.nameservices/name and each nameservice will have two namenodes. If i configure only HA (only one nameservice), everything is ok, and HAAdmin can determine the namenodes nn1, nn3. But if i configure two nameservice and set namenodes nn1,nn3 for nameservice1 and nn2,nn4 for nameservices2. I can start these namenodes successfully and the namenodes are all in standby state at th beginning. But if i want to change one namenode to active state, use command hdfs haadmin -transitionToActive nn1 HAAdmin throw exception as it cannot determine the four namenodes(nn1,nn2,nn3,nn4) at all. Do you used to configure HAFederation and know what may cause these problem? Thanks, Lucy -- Original -- From: Matt Narrell;matt.narr...@gmail.com; Send time: Monday, Sep 29, 2014 6:28 AM To: useruser@hadoop.apache.org; Subject: Re: Failed to active namenode when config HA I’m pretty sure HDFS HA is relegated to two name nodes (not four), designated active and standby. Secondly, I believe these properties should be in hdfs-site.xml NOT core-site.xml. Furthermore, I think your HDFS nameservices are misconfigured. Consider the following: ?xml version=1.0? configuration property namedfs.replication/name value3/value /property property namedfs.namenode.name.dir/name valuefile:/var/data/hadoop/hdfs/nn/value /property property namedfs.datanode.data.dir/name valuefile:/var/data/hadoop/hdfs/dn/value /property property namedfs.ha.automatic-failover.enabled/name valuetrue/value /property property namedfs.nameservices/name valuehdfs-cluster/value /property property namedfs.ha.namenodes.hdfs-cluster/name valuenn1,nn2/value /property property namedfs.namenode.rpc-address.hdfs-cluster.nn1/name valuenamenode1:8020/value /property property
