Does your DN log show up any form of errors when you run into this?
This happens with just two jobs reading how many files? And how many
DNs are these spread across?
I'm thinking its probably something to do with your ulimits for the
running DN processes, but I can't say for sure without taking a
We just reproduced it (somehow) by running jobs concurrently reading the same
data. Two out of three similar jobs died early in the map phase with Could not
obtain block errors, one finished completely.
java.io.IOException: Could not obtain block: blk_119146860335302651_13067
file=/user/system
To follow up on what I have found:
I opened up some of the logs on the datanodes and found this message:
"Can not start task tracker because java.net.BindException: Address
already in use"
It was using the default port setting from mapred-default.xml, which was 50060.
I decided to try an add
1) My guess is that was included in one of the patches ported to the
0.22 branch.
2) You're correct, you need to limit access to the cluster via a
gateway. This isn't really sufficient as the code in your MapReduce
job will run as the mapred user and have access to anything it can
see.
If you car
Joey,
1) if(!conf.get(HADOOP_SECURITY_AUTHENTICATION).equals("kerberos"))
throw new RuntimeException("Cannot start secure datanode in unsecure
cluster");
Then what is this code in
o.a.h.hdfs.server.datanode.SecureDataNodeStarter.java about?
2) If Kerberos is not supported, then how to auth
Yes, it means that 0.22 doesn't support Kerberos.
-Joey
On Thu, Dec 29, 2011 at 9:41 AM, Praveen Sripati
wrote:
> Hi,
>
> The release notes for 0.22
> (http://hadoop.apache.org/common/releases.html#10+December%2C+2011%3A+release+0.22.0+available)
> it says
>
>>The following features are not supp
Hi,
The release notes for 0.22 (
http://hadoop.apache.org/common/releases.html#10+December%2C+2011%3A+release+0.22.0+available)
it says
>The following features are not supported in Hadoop 0.22.0.
>Security.
>Latest optimizations of the MapReduce framework introduced in the
Hadoop 0.20.se
Hi, I use Yarn as resource management to deploy my run-time computing
system. I follow
http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/YARN.html
>>
>> http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
>>
> as guide, and
