Hadoop 2.2.0 YARN isolation and Windows

[Kevin]

Hi,

Three questions about the new Hadoop release regarding YARN:

1. Does YARN need to run on the same machines that are hosting the HDFS
services or can HDFS be remote of a YARN cluster? If this done by placing
the remote HDFS cluster's configuration files (core-site.xml and
hdfs-site.xml) on the YARN cluster's machines?

2. According to
http://www.i-programmer.info/news/197-data-mining/6518-hadoop-2-introduces-yarn.html,
Hadoop 2.2.0 supports Microsoft Windows. How do/Can you configure YARN for
secure container isolation in Windows? It seems that the ContainerExecutor
and DefaultContainerExecutor can detect and run on Windows, but the secure
LinuxContainerExecutor are for *nix systems, so is there anything in place
for maximum security like LCE is?

3. If 1 is yes, then is it possible to have a cluster mixed with both Linux
and Windows machines running YARN and working together?

Thanks,
Kevin

Re: run arbitrary job (non-MR) on YARN ?

[Kevin]

You can accomplish this by using the DistributedShell application that
comes with YARN.

If you copy all your archives to HDFS, then inside your shell script you
could copy those archives to your YARN container and then execute whatever
you want, provided all the other system dependencies exist in the container
(correct Java version, Python, C++ libraries, etc.)

For example,

In myscript.sh I wrote the following:

#!/usr/bin/env bash
echo "This is my script running!"
echo "Present working directory:"
pwd
echo "Current directory listing: (nothing exciting yet)"
ls
echo "Copying file from HDFS to container"
hadoop fs -get /path/to/some/data/on/hdfs .
echo "Current directory listing: (file should not be here)"
ls
echo "Cat ExecScript.sh (this is the script created by the DistributedShell
application)"
cat ExecScript.sh

Run the DistributedShell application with the hadoop (or yarn) command:

hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.3.0-cdh5.1.3.jar
-num_containers 1 -shell_script myscript.sh

If you have the YARN log aggregation property set, then you can pipe the
container's logs to your client console using the yarn command:

yarn logs -applicationId application_1414160538995_0035

(replace the application id with yours)

Here is a quick reference that should help get you going:
http://books.google.com/books?id=heoXAwAAQBAJ&pg=PA227&lpg=PA227&dq=hadoop+yarn+distributed+shell+application&source=bl&ots=psGuJYlY1Y&sig=khp3b3hgzsZLZWFfz7GOe2yhgyY&hl=en&sa=X&ei=0U5RVKzDLeTK8gGgoYGoDQ&ved=0CFcQ6AEwCA#v=onepage&q&f=false

Hopefully this helps,
Kevin

On Mon Oct 27 2014 at 2:21:18 AM Yang  wrote:

> I happened to run into this interesting scenario:
>
> I had some mahout seq2sparse jobs, originally i run them in parallel using
> the distributed mode. but because the input files are so small, running
> them locally actually is much faster. so I truned them to local mode.
>
> but I run 10 of these jobs in parallel, so when 10 mahout jobs are run
> together, everyone became very slow.
>
> is there an existing code that takes a desired shell script, and possibly
> some archive files (could contain the jar file, or C++ --generated
> executable code). I understand that I could use yarn API to code such a
> thing, but it would be nice if I could just take it and run in shell..
>
> Thanks
> Yang
>

Permissions issue with launching MR job from Oozie shell

[Kevin]

I hope this question is relevant to the Hadoop mailing list. If it should
be asked to the Oozie community, then I apologize in advance.

I'm trying to schedule an HBase MapReduce bulkload preparation job with
Oozie. I need to use either the Oozie shell action or Java action since the
driver class needs to create the partition file, determine the number of
reducers, etc. I chose the shell action as my solution.

As user 'kevin', I submit and run my Oozie workflow (using the oozie client
command). I understand that Oozie executes the shell as the yarn user, but
it appears that the user 'kevin' in being used to access the yarn staging
directory. The application master's container logs are below.

My main question is:
Why is the user 'kevin' being used to run the application master? (Or maybe
it's not and I don't even understand that much)

I am using CDH5.1.3 with YARN.

The script is simple:

#!/usr/bin/env bash
hadoop jar myjar.jar package.Driver -i $INPUT -o $OUTPUT_DIR

In mapred-site.xml:


  yarn.app.mapreduce.am.staging-dir
  /user


The error I am getting is:

2014-11-17 12:43:31,428 INFO [main]
org.apache.hadoop.mapreduce.JobSubmitter: Kind: RM_DELEGATION_TOKEN,
Service: 10.10.23.111:8032,10.10.23.112:8032, Ident: (owner=kevin,
renewer=oozie mr token, realUser=oozie, issueDate=1416246198904,
maxDate=1416850998904, sequenceNumber=162, masterKeyId=87)
...
...
...
2014-11-17 12:43:34,905 INFO [main] org.apache.hadoop.mapreduce.Job: Job
job_1415194788406_0050 failed with state FAILED due to: Application
application_1415194788406_0050 failed 2 times due to AM Container for
appattempt_1415194788406_0050_02 exited with  exitCode: -1000 due to:
Permission denied: user=kevin, access=EXECUTE,
inode="/user/yarn":yarn:hadoop:drwx--
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:257)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:208)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:171)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5607)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3583)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:766)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:764)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)

Re: Run a c++ program using opencv libraries in hadoop

[Kevin]

You could run it as a shell action using Oozie. Write a shell script to run
your application. Put all the application's dependencies (e.g., *.so) into
a lib directory. Put the shell script in the parent directory of that lib
directory that I just mentioned. Create a simple Oozie workflow that runs
that shell script. Save the workflow.xml that you just created to put the
shell script in. Finally, make a directory on HDFS and put all of it in
that directory. Oozie will run that shell script in a single-map,
no-reducer job.

On Wed, Dec 17, 2014 at 11:12 PM, unmesha sreeveni 
wrote:
>
> Hi
>
>  How can I run c++ programs using opencv libraries in hadoop?
>
> So far I have done MapReduce jobs in Java only..and there we can supply
> external jars using command line itself.
> And even tried using python language also..to run them we use hadoop
> streaming API.
> But I am confused how to run C++ programs using opencv libraries.
>
> Thanks in Advance.
>
> --
> *Thanks & Regards *
>
>
> *Unmesha Sreeveni U.B*
> *Hadoop, Bigdata Developer*
> *Centre for Cyber Security | Amrita Vishwa Vidyapeetham*
> http://www.unmeshasreeveni.blogspot.in/
>
>
>

Copying many files to HDFS

[Kevin]

Hi,

I am setting up a Hadoop cluster (CDH5.1.3) and I need to copy a thousand
or so files into HDFS, which totals roughly 1 TB. The cluster will be
isolated on its own private LAN with a single client machine that is
connected to the Hadoop cluster as well as the public network. The data
that needs to be copied into HDFS is mounted as an NFS on the client
machine.

I can run `hadoop fs -put` concurrently on the client machine to try and
increase the throughput.

If these files were able to be accessed by each node in the Hadoop cluster,
then I could write a MapReduce job to copy a number of files from the
network into HDFS. I could not find anything in the documentation saying
that `distcp` works with locally hosted files (its code in the tools
package doesn't tell any sign of it either) - but I wouldn't expect it to.

In general, are there any other ways of copying a very large number of
client-local files to HDFS? I search the mail archives to find a similar
question and I didn't come across one. I'm sorry if this is a duplicate
question.

Thanks for your time,
Kevin

Re: Copying many files to HDFS

[Kevin]

Ahmed,
Flume is a great tool but it doesn't cover my use case. I need to copy the
files in their entirety and keep their file names.

Alexander,
Thanks for sharing Slurper. From the code it looks like a reasonable
multi-threaded application to copy files. I'll keep looking at it.

On Fri, Feb 13, 2015 at 9:03 AM, Alexander Alten-Lorenz  wrote:

> Kevin,
>
> Slurper can help here:
> https://github.com/alexholmes/hdfs-file-slurper
>
> BR,
>  Alexander
>
>
> On 13 Feb 2015, at 14:28, Kevin  wrote:
>
> Hi,
>
> I am setting up a Hadoop cluster (CDH5.1.3) and I need to copy a thousand
> or so files into HDFS, which totals roughly 1 TB. The cluster will be
> isolated on its own private LAN with a single client machine that is
> connected to the Hadoop cluster as well as the public network. The data
> that needs to be copied into HDFS is mounted as an NFS on the client
> machine.
>
> I can run `hadoop fs -put` concurrently on the client machine to try and
> increase the throughput.
>
> If these files were able to be accessed by each node in the Hadoop
> cluster, then I could write a MapReduce job to copy a number of files from
> the network into HDFS. I could not find anything in the documentation
> saying that `distcp` works with locally hosted files (its code in the tools
> package doesn't tell any sign of it either) - but I wouldn't expect it to.
>
> In general, are there any other ways of copying a very large number of
> client-local files to HDFS? I search the mail archives to find a similar
> question and I didn't come across one. I'm sorry if this is a duplicate
> question.
>
> Thanks for your time,
> Kevin
>
>
>

Re: Copying many files to HDFS

[Kevin]

Johny, NiFi looks interesting but I can't really grasp how it will help me.
If you could provided some example code or a more detail explanation of how
you set up a topology, then that would be great.

On Fri, Feb 13, 2015 at 10:38 AM, johny casanova 
wrote:

>
>  Hi Kevin,
>
> You can try Apache nifi https://nifi.incubator.apache.org/ is a new
> application that is still in incubation but, awesome tool to use for what
> you are looking for. Ithas a processor that put data and get data from HDFS
> and send continuously without having to use the put command. Check them out
> and let me know if you need help. I use it to put to HDFS also and put high
> volumes like you mentioned.
>  --
> Date: Fri, 13 Feb 2015 09:25:35 -0500
> Subject: Re: Copying many files to HDFS
> From: kevin.macksa...@gmail.com
> To: user@hadoop.apache.org
>
>
> Ahmed,
> Flume is a great tool but it doesn't cover my use case. I need to copy the
> files in their entirety and keep their file names.
>
> Alexander,
> Thanks for sharing Slurper. From the code it looks like a reasonable
> multi-threaded application to copy files. I'll keep looking at it.
>
> On Fri, Feb 13, 2015 at 9:03 AM, Alexander Alten-Lorenz <
> wget.n...@gmail.com> wrote:
>
>  Kevin,
>
> Slurper can help here:
> https://github.com/alexholmes/hdfs-file-slurper
>
> BR,
>  Alexander
>
>
>  On 13 Feb 2015, at 14:28, Kevin  wrote:
>
>  Hi,
>
> I am setting up a Hadoop cluster (CDH5.1.3) and I need to copy a thousand
> or so files into HDFS, which totals roughly 1 TB. The cluster will be
> isolated on its own private LAN with a single client machine that is
> connected to the Hadoop cluster as well as the public network. The data
> that needs to be copied into HDFS is mounted as an NFS on the client
> machine.
>
> I can run `hadoop fs -put` concurrently on the client machine to try and
> increase the throughput.
>
> If these files were able to be accessed by each node in the Hadoop
> cluster, then I could write a MapReduce job to copy a number of files from
> the network into HDFS. I could not find anything in the documentation
> saying that `distcp` works with locally hosted files (its code in the tools
> package doesn't tell any sign of it either) - but I wouldn't expect it to.
>
> In general, are there any other ways of copying a very large number of
> client-local files to HDFS? I search the mail archives to find a similar
> question and I didn't come across one. I'm sorry if this is a duplicate
> question.
>
> Thanks for your time,
> Kevin
>
>
>
>

Lifetime of jhist files

[Kevin]

Hi,

I am running CDH5.1.3 with YARN. The one issue I am having is that the
jhist files are being deleted too quickly. I
set yarn.nodemanager.delete.debug-delay-sec to seven days (in seconds, of
course), but I am noticing that the job historyserver is expiring any jhist
file that is seven days old. I thought
yarn.nodemanager.delete.debug-delay-sec was just to clean up local logs on
the NodeManager; I didn't think it also affected the job history files too.

Am I right in my assumptions? Is there anyway to extend the lifetime of the
job history files?

BTW, my history files are stored in /user/history/done///

Any feedback would be great.

Thanks,
Kevin

Re: Lifetime of jhist files

[Kevin]

Thanks, Naga, that worked. I didn't catch that property in the
mapred-default.xml. Sorry for such a late response.

On Tue, Apr 28, 2015 at 11:01 PM, Naganarasimha G R (Naga) <
garlanaganarasi...@huawei.com> wrote:

>  Hi Kevin,
>
>  Could check the below configuration for the job history server
>   mapreduce.jobhistory.max-age-ms 60480 Job history files older than
> this many milliseconds will be deleted when the history cleaner runs.
> Defaults to 60480 (1 week).
> you could refer other job history configs in* 
> http://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
> <http://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml>*
>
> +Naga
>
>  --
> *From:* Kevin [kevin.macksa...@gmail.com]
> *Sent:* Wednesday, April 29, 2015 01:11
> *To:* user@hadoop.apache.org
> *Subject:* Lifetime of jhist files
>
>   Hi,
>
>  I am running CDH5.1.3 with YARN. The one issue I am having is that the
> jhist files are being deleted too quickly. I
> set yarn.nodemanager.delete.debug-delay-sec to seven days (in seconds, of
> course), but I am noticing that the job historyserver is expiring any jhist
> file that is seven days old. I thought
> yarn.nodemanager.delete.debug-delay-sec was just to clean up local logs on
> the NodeManager; I didn't think it also affected the job history files too.
>
>  Am I right in my assumptions? Is there anyway to extend the lifetime of
> the job history files?
>
>  BTW, my history files are stored in
> /user/history/done///
>
>  Any feedback would be great.
>
>  Thanks,
> Kevin
>

Using YARN with native applications

[Kevin]

Hello,

I have been using the distributed shell application and Oozie to run native
C++ applications in the cluster. Is YARN able to see the resources these
native applications use. For example, if I use Oozie's shell action, the
NodeManager hosts the mapper container and allocates a certain amount of
memory and vcores (as configured). What happens if my C++ application uses
more memory or vcores than the NodeManager allocated?

I was looking in the Hadoop code and I couldn't find my way to answer.
Although, it seems the LinuxContainerExecutor may be the answer to my
question since it uses cgroups.

I'm interested to know how YARN reacts to non-Java applications running
inside of it.

Thanks,
Kevin

Re: Using YARN with native applications

[Kevin]

Thanks for the reply, Varun. So if I use the DefaultContainerExecutor and
run a C++ application via a shell script inside a container whose virtual
memory limit is, for example, 2 GB, and that application does a malloc for
3 GB, YARN will kill the container? I always just thought that YARN kept
its eye on the JVM it spins up for the container (under the
DefaultContainerExecutor).

-Kevin

On Mon, May 25, 2015 at 4:17 AM, Varun Vasudev 
wrote:

>  Hi Kevin,
>
>  By default, the NodeManager monitors physical and virtual memory usage
> of containers. Containers that exceed either limit are killed. Admins can
> disable the checks by setting yarn.nodemanager.pmem-check-enabled and/or 
> yarn.nodemanager.vmem-check-enabled
> to false. The virtual memory limit for a container is determined using the
> config variable yarn.nodemanager.vmem-pmem-ratio(default value is 2.1).
>
>  In case of vcores -
>
>1. If you’re using Cgroups under LinuxContainerExecutor, by default,
>if there is spare CPU available on the node, your container will be allowed
>to use it. Admins can restrict containers to use only the CPU allocated to
>them by setting 
> yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage
>to true. This setting is only applicable when using Cgroups under
>LinuxContainerExecutor.
>2.  If you aren’t using Cgroups under LinuxContainerExecutor, there is
>no limiting of the amount of the CPU that containers can use.
>
>  -Varun
>
>   From: Kevin
> Reply-To: "user@hadoop.apache.org"
> Date: Friday, May 22, 2015 at 3:30 AM
> To: "user@hadoop.apache.org"
> Subject: Using YARN with native applications
>
>   Hello,
>
>  I have been using the distributed shell application and Oozie to run
> native C++ applications in the cluster. Is YARN able to see the resources
> these native applications use. For example, if I use Oozie's shell action,
> the NodeManager hosts the mapper container and allocates a certain amount
> of memory and vcores (as configured). What happens if my C++ application
> uses more memory or vcores than the NodeManager allocated?
>
>  I was looking in the Hadoop code and I couldn't find my way to answer.
> Although, it seems the LinuxContainerExecutor may be the answer to my
> question since it uses cgroups.
>
>  I'm interested to know how YARN reacts to non-Java applications running
> inside of it.
>
>  Thanks,
> Kevin
>

Re: Using YARN with native applications

[Kevin]

Varun, thank you for helping me understand this. You pointed out a couple
of new things to me. I finally found that monitoring thread in the code
(ContainersMonitorImpl.java). I can now see and gain a better understanding
of YARN checks on a container's resources.

On Wed, May 27, 2015 at 1:23 AM Varun Vasudev 
wrote:

>  YARN should kill the container. I’m not sure what JVM you’re referring
> to, but the NodeManager writes and then spawns a shell script that will
> invoke your shell script which in turn(presumably) will invoke your C++
> application. A monitoring thread then looks at the memory usage of the
> process tree and compares it to the limits for the container.
>
>  -Varun
>
>   From: Kevin
> Reply-To: "user@hadoop.apache.org"
> Date: Tuesday, May 26, 2015 at 7:22 AM
> To: "user@hadoop.apache.org"
> Subject: Re: Using YARN with native applications
>
>   Thanks for the reply, Varun. So if I use the DefaultContainerExecutor
> and run a C++ application via a shell script inside a container whose
> virtual memory limit is, for example, 2 GB, and that application does a
> malloc for 3 GB, YARN will kill the container? I always just thought that
> YARN kept its eye on the JVM it spins up for the container (under the
> DefaultContainerExecutor).
>
>  -Kevin
>
> On Mon, May 25, 2015 at 4:17 AM, Varun Vasudev 
> wrote:
>
>>  Hi Kevin,
>>
>>  By default, the NodeManager monitors physical and virtual memory usage
>> of containers. Containers that exceed either limit are killed. Admins can
>> disable the checks by setting yarn.nodemanager.pmem-check-enabled and/or 
>> yarn.nodemanager.vmem-check-enabled
>> to false. The virtual memory limit for a container is determined using the
>> config variable yarn.nodemanager.vmem-pmem-ratio(default value is 2.1).
>>
>>  In case of vcores -
>>
>>1. If you’re using Cgroups under LinuxContainerExecutor, by default,
>>if there is spare CPU available on the node, your container will be 
>> allowed
>>to use it. Admins can restrict containers to use only the CPU allocated to
>>them by setting 
>> yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage
>>to true. This setting is only applicable when using Cgroups under
>>LinuxContainerExecutor.
>>2.  If you aren’t using Cgroups under LinuxContainerExecutor, there
>>is no limiting of the amount of the CPU that containers can use.
>>
>>  -Varun
>>
>>   From: Kevin
>> Reply-To: "user@hadoop.apache.org"
>> Date: Friday, May 22, 2015 at 3:30 AM
>> To: "user@hadoop.apache.org"
>> Subject: Using YARN with native applications
>>
>>   Hello,
>>
>>  I have been using the distributed shell application and Oozie to run
>> native C++ applications in the cluster. Is YARN able to see the resources
>> these native applications use. For example, if I use Oozie's shell action,
>> the NodeManager hosts the mapper container and allocates a certain amount
>> of memory and vcores (as configured). What happens if my C++ application
>> uses more memory or vcores than the NodeManager allocated?
>>
>>  I was looking in the Hadoop code and I couldn't find my way to answer.
>> Although, it seems the LinuxContainerExecutor may be the answer to my
>> question since it uses cgroups.
>>
>>  I'm interested to know how YARN reacts to non-Java applications running
>> inside of it.
>>
>>  Thanks,
>> Kevin
>>
>
>

Re: Using YARN with native applications

[Kevin]

Thanks for the tip. In the trunk it looks like the NodeManager's monitor
thread doesn't care if the process tree's cores overflows the container's
CPU limit. Is this monitored elsewhere?

I have my eyes on
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L476


On Wed, May 27, 2015 at 9:06 AM Varun Vasudev 
wrote:

>   You should also look at ProcfsBasedProcessTree if you want to know how
> exactly the memory usage is being calculated.
>
>  -Varun
>
>   From: Kevin
> Reply-To: "user@hadoop.apache.org"
> Date: Wednesday, May 27, 2015 at 6:22 PM
>
> To: "user@hadoop.apache.org"
> Subject: Re: Using YARN with native applications
>
>   Varun, thank you for helping me understand this. You pointed out a
> couple of new things to me. I finally found that monitoring thread in the
> code (ContainersMonitorImpl.java). I can now see and gain a better
> understanding of YARN checks on a container's resources.
>
>  On Wed, May 27, 2015 at 1:23 AM Varun Vasudev 
> wrote:
>
>>  YARN should kill the container. I’m not sure what JVM you’re referring
>> to, but the NodeManager writes and then spawns a shell script that will
>> invoke your shell script which in turn(presumably) will invoke your C++
>> application. A monitoring thread then looks at the memory usage of the
>> process tree and compares it to the limits for the container.
>>
>>  -Varun
>>
>>   From: Kevin
>> Reply-To: "user@hadoop.apache.org"
>> Date: Tuesday, May 26, 2015 at 7:22 AM
>> To: "user@hadoop.apache.org"
>> Subject: Re: Using YARN with native applications
>>
>>   Thanks for the reply, Varun. So if I use the DefaultContainerExecutor
>> and run a C++ application via a shell script inside a container whose
>> virtual memory limit is, for example, 2 GB, and that application does a
>> malloc for 3 GB, YARN will kill the container? I always just thought that
>> YARN kept its eye on the JVM it spins up for the container (under the
>> DefaultContainerExecutor).
>>
>>  -Kevin
>>
>> On Mon, May 25, 2015 at 4:17 AM, Varun Vasudev 
>> wrote:
>>
>>>  Hi Kevin,
>>>
>>>  By default, the NodeManager monitors physical and virtual memory usage
>>> of containers. Containers that exceed either limit are killed. Admins can
>>> disable the checks by setting yarn.nodemanager.pmem-check-enabled
>>> and/or yarn.nodemanager.vmem-check-enabled to false. The virtual memory
>>> limit for a container is determined using the config variable 
>>> yarn.nodemanager.vmem-pmem-ratio(default
>>> value is 2.1).
>>>
>>>  In case of vcores -
>>>
>>>1. If you’re using Cgroups under LinuxContainerExecutor, by default,
>>>if there is spare CPU available on the node, your container will be 
>>> allowed
>>>to use it. Admins can restrict containers to use only the CPU allocated 
>>> to
>>>them by setting 
>>> yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage
>>>to true. This setting is only applicable when using Cgroups under
>>>LinuxContainerExecutor.
>>>2.  If you aren’t using Cgroups under LinuxContainerExecutor, there
>>>is no limiting of the amount of the CPU that containers can use.
>>>
>>>  -Varun
>>>
>>>   From: Kevin
>>> Reply-To: "user@hadoop.apache.org"
>>> Date: Friday, May 22, 2015 at 3:30 AM
>>> To: "user@hadoop.apache.org"
>>> Subject: Using YARN with native applications
>>>
>>>   Hello,
>>>
>>>  I have been using the distributed shell application and Oozie to run
>>> native C++ applications in the cluster. Is YARN able to see the resources
>>> these native applications use. For example, if I use Oozie's shell action,
>>> the NodeManager hosts the mapper container and allocates a certain amount
>>> of memory and vcores (as configured). What happens if my C++ application
>>> uses more memory or vcores than the NodeManager allocated?
>>>
>>>  I was looking in the Hadoop code and I couldn't find my way to answer.
>>> Although, it seems the LinuxContainerExecutor may be the answer to my
>>> question since it uses cgroups.
>>>
>>>  I'm interested to know how YARN reacts to non-Java applications
>>> running inside of it.
>>>
>>>  Thanks,
>>> Kevin
>>>
>>
>>

Re: Using YARN with native applications

[Kevin]

Ah, okay. That makes sense. Thanks for all your help, Varun.

-Kevin

On Wed, May 27, 2015 at 9:53 AM Varun Vasudev 
wrote:

>   For CPU isolation, you have to use Cgroups with the
> LinuxContainerExecutor. We don’t enforce cpu limits with the
> DefaultContainerExecutor.
>
>  -Varun
>
>   From: Kevin
> Reply-To: "user@hadoop.apache.org"
> Date: Wednesday, May 27, 2015 at 7:06 PM
>
> To: "user@hadoop.apache.org"
> Subject: Re: Using YARN with native applications
>
>   Thanks for the tip. In the trunk it looks like the NodeManager's
> monitor thread doesn't care if the process tree's cores overflows the
> container's CPU limit. Is this monitored elsewhere?
>
>  I have my eyes on
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L476
>
>
>  On Wed, May 27, 2015 at 9:06 AM Varun Vasudev 
> wrote:
>
>>   You should also look at ProcfsBasedProcessTree if you want to know how
>> exactly the memory usage is being calculated.
>>
>>  -Varun
>>
>>   From: Kevin
>> Reply-To: "user@hadoop.apache.org"
>> Date: Wednesday, May 27, 2015 at 6:22 PM
>>
>> To: "user@hadoop.apache.org"
>> Subject: Re: Using YARN with native applications
>>
>>   Varun, thank you for helping me understand this. You pointed out a
>> couple of new things to me. I finally found that monitoring thread in the
>> code (ContainersMonitorImpl.java). I can now see and gain a better
>> understanding of YARN checks on a container's resources.
>>
>>  On Wed, May 27, 2015 at 1:23 AM Varun Vasudev 
>> wrote:
>>
>>>  YARN should kill the container. I’m not sure what JVM you’re referring
>>> to, but the NodeManager writes and then spawns a shell script that will
>>> invoke your shell script which in turn(presumably) will invoke your C++
>>> application. A monitoring thread then looks at the memory usage of the
>>> process tree and compares it to the limits for the container.
>>>
>>>  -Varun
>>>
>>>   From: Kevin
>>> Reply-To: "user@hadoop.apache.org"
>>> Date: Tuesday, May 26, 2015 at 7:22 AM
>>> To: "user@hadoop.apache.org"
>>> Subject: Re: Using YARN with native applications
>>>
>>>   Thanks for the reply, Varun. So if I use the DefaultContainerExecutor
>>> and run a C++ application via a shell script inside a container whose
>>> virtual memory limit is, for example, 2 GB, and that application does a
>>> malloc for 3 GB, YARN will kill the container? I always just thought that
>>> YARN kept its eye on the JVM it spins up for the container (under the
>>> DefaultContainerExecutor).
>>>
>>>  -Kevin
>>>
>>> On Mon, May 25, 2015 at 4:17 AM, Varun Vasudev >> > wrote:
>>>
>>>>  Hi Kevin,
>>>>
>>>>  By default, the NodeManager monitors physical and virtual memory
>>>> usage of containers. Containers that exceed either limit are killed. Admins
>>>> can disable the checks by setting yarn.nodemanager.pmem-check-enabled
>>>> and/or yarn.nodemanager.vmem-check-enabled to false. The virtual
>>>> memory limit for a container is determined using the config variable 
>>>> yarn.nodemanager.vmem-pmem-ratio(default
>>>> value is 2.1).
>>>>
>>>>  In case of vcores -
>>>>
>>>>1. If you’re using Cgroups under LinuxContainerExecutor, by
>>>>default, if there is spare CPU available on the node, your container 
>>>> will
>>>>be allowed to use it. Admins can restrict containers to use only the CPU
>>>>allocated to them by setting 
>>>> yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage
>>>>to true. This setting is only applicable when using Cgroups under
>>>>LinuxContainerExecutor.
>>>>2.  If you aren’t using Cgroups under LinuxContainerExecutor, there
>>>>is no limiting of the amount of the CPU that containers can use.
>>>>
>>>>  -Varun
>>>>
>>>>   From: Kevin
>>>> Reply-To: "user@hadoop.apache.org"
>>>> Date: Friday, May 22, 2015 at 3:30 AM
>>>> To: "user@hadoop.apache.org"
>>>> Subject: Using YARN with native applications
>>>>
>>>>   Hello,
>>>>
>>>>  I have been using the distributed shell application and Oozie to run
>>>> native C++ applications in the cluster. Is YARN able to see the resources
>>>> these native applications use. For example, if I use Oozie's shell action,
>>>> the NodeManager hosts the mapper container and allocates a certain amount
>>>> of memory and vcores (as configured). What happens if my C++ application
>>>> uses more memory or vcores than the NodeManager allocated?
>>>>
>>>>  I was looking in the Hadoop code and I couldn't find my way to
>>>> answer. Although, it seems the LinuxContainerExecutor may be the answer to
>>>> my question since it uses cgroups.
>>>>
>>>>  I'm interested to know how YARN reacts to non-Java applications
>>>> running inside of it.
>>>>
>>>>  Thanks,
>>>> Kevin
>>>>
>>>
>>>

About Archival Storage

[kevin]

I don't quite understand :"Note that the Lazy_Persist policy is useful only
for single replica blocks. For blocks with more than one replicas, all the
replicas will be written to DISK since writing only one of the replicas to
RAM_DISK does not improve the overall performance."

Is that mean I should config dfs.replication with 1 ?  if more than one I
should not use *Lazy_Persist*  policies ?

Re: About Archival Storage

[kevin]

Thanks a lot Rakesh.

I have another question is , hdfs mover (A New Data Migration Tool ) know
when to move data from hot to cold  automatically ? It use algorithm
like LRU、LFU ?

2016-07-19 19:55 GMT+08:00 Rakesh Radhakrishnan :

> >>>>Is that mean I should config dfs.replication with 1 ?  if more than
> one I should not use *Lazy_Persist*  policies ?
>
> The idea of Lazy_Persist policy is, while writing blocks, one replica will
> be placed in memory first and then it is lazily persisted into DISK. It
> doesn't means that, you are not allowed to configure dfs.replication > 1.
> If 'dfs.replication' is configured > 1 then the first replica will be
> placed in RAM_DISK and all the other replicas (n-1) will be written to the
> DISK. Here the (n-1) replicas will have the overhead of pipeline
> replication over the network and the DISK write latency on the write hot
> path. So you will not get better performance results.
>
> IIUC, for getting memory latency benefits, it is recommended to use
> replication=1. In this way, applications should be able to perform single
> replica writes to a local DN with low latency. HDFS will store block data
> in memory and lazily save it to disk avoiding incurring disk write latency
> on the hot path. By writing to local memory we can also avoid checksum
> computation on the hot path.
>
> Regards,
> Rakesh
>
> On Tue, Jul 19, 2016 at 3:25 PM, kevin  wrote:
>
>> I don't quite understand :"Note that the Lazy_Persist policy is useful
>> only for single replica blocks. For blocks with more than one replicas, all
>> the replicas will be written to DISK since writing only one of the replicas
>> to RAM_DISK does not improve the overall performance."
>>
>> Is that mean I should config dfs.replication with 1 ?  if more than one I
>> should not use *Lazy_Persist*  policies ?
>>
>
>

Re: About Archival Storage

[kevin]

Thanks again. "automatically" what I mean is the hdfs mover knows the hot
data have come to cold , I don't need to tell it what exactly files/dirs
need to be move now ?
Of course I should tell it what files/dirs need to monitoring.

2016-07-20 12:35 GMT+08:00 Rakesh Radhakrishnan :

> >>>I have another question is , hdfs mover (A New Data Migration Tool )
> know when to move data from hot to cold  automatically ?
> While running the tool, it reads the argument and get the separated list
> of hdfs files/dirs to migrate. Then it periodically scans these files in
> HDFS to check if the block placement satisfies the storage policy, if not
> satisfied it moves the replicas to a different storage type in order to
> fulfill the storage policy requirement. This cycle continues until it hits
> an error or no blocks to move etc. Could you please tell me, what do you
> meant by "automatically" ? FYI, HDFS-10285 is proposing an idea to
> introduce a daemon thread in Namenode to track the storage movements set by
> APIs from clients. This Daemon thread named as StoragePolicySatisfier(SPS)
> serves something similar to ReplicationMonitor. If interested you can read
> the https://goo.gl/NA5EY0 proposal/idea and welcome feedback.
>
> Sleep time between each cycle is, ('dfs.heartbeat.interval' * 2000) +
> ('dfs.namenode.replication.interval' * 1000) milliseconds;
>
> >>>It use algorithm like LRU、LFU ?
> It will simply iterating over the lists in the order of files/dirs given
> to this tool as an argument. afaik, its just maintains the order mentioned
> by the user.
>
> Regards,
> Rakesh
>
>
> On Wed, Jul 20, 2016 at 7:05 AM, kevin  wrote:
>
>> Thanks a lot Rakesh.
>>
>> I have another question is , hdfs mover (A New Data Migration Tool )
>> know when to move data from hot to cold  automatically ? It
>> use algorithm like LRU、LFU ?
>>
>> 2016-07-19 19:55 GMT+08:00 Rakesh Radhakrishnan :
>>
>>> >>>>Is that mean I should config dfs.replication with 1 ?  if more than
>>> one I should not use *Lazy_Persist*  policies ?
>>>
>>> The idea of Lazy_Persist policy is, while writing blocks, one replica
>>> will be placed in memory first and then it is lazily persisted into DISK.
>>> It doesn't means that, you are not allowed to configure dfs.replication >
>>> 1. If 'dfs.replication' is configured > 1 then the first replica will be
>>> placed in RAM_DISK and all the other replicas (n-1) will be written to the
>>> DISK. Here the (n-1) replicas will have the overhead of pipeline
>>> replication over the network and the DISK write latency on the write hot
>>> path. So you will not get better performance results.
>>>
>>> IIUC, for getting memory latency benefits, it is recommended to use
>>> replication=1. In this way, applications should be able to perform single
>>> replica writes to a local DN with low latency. HDFS will store block data
>>> in memory and lazily save it to disk avoiding incurring disk write latency
>>> on the hot path. By writing to local memory we can also avoid checksum
>>> computation on the hot path.
>>>
>>> Regards,
>>> Rakesh
>>>
>>> On Tue, Jul 19, 2016 at 3:25 PM, kevin  wrote:
>>>
>>>> I don't quite understand :"Note that the Lazy_Persist policy is useful
>>>> only for single replica blocks. For blocks with more than one replicas, all
>>>> the replicas will be written to DISK since writing only one of the replicas
>>>> to RAM_DISK does not improve the overall performance."
>>>>
>>>> Is that mean I should config dfs.replication with 1 ?  if more than one
>>>> I should not use *Lazy_Persist*  policies ?
>>>>
>>>
>>>
>>
>

Re: About Archival Storage

[kevin]

Ok,thanks.It means that I need to decide which data is hot and which time
it is cold, then change it storage policy and tell  'Mover tool'  to move
it.

2016-07-20 14:29 GMT+08:00 Rakesh Radhakrishnan :

> Based on storage policy the data from hot storage will be moved to cold
> storage. The storage policy defines the number of replicas to be located on
> each storage type. It is possible to change the storage policy on a
> directory(for example: HOT to COLD) and then invoke 'Mover tool' on that
> directory to make the policy effective. One can set/change the storage
> policy via HDFSCommand, "hdfs storagepolicies -setStoragePolicy -path
>  -policy ". After setting the new policy, you need to run the
> tool, then it identifies the replicas to be moved based on the storage
> policy information, and schedules the movement between source and
> destination data nodes to satisfy the policy. Internally, the tool is
> comparing the 'storage type' of a block in order to fulfill the 'storage
> policy' requirement.
>
> Probably you can refer
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html
> to know more about storage types, storage policies and hdfs commands. Hope
> this helps.
>
> Rakesh
>
> On Wed, Jul 20, 2016 at 10:30 AM, kevin  wrote:
>
>> Thanks again. "automatically" what I mean is the hdfs mover knows the
>> hot data have come to cold , I don't need to tell it what exactly files/dirs
>> need to be move now ?
>> Of course I should tell it what files/dirs need to monitoring.
>>
>> 2016-07-20 12:35 GMT+08:00 Rakesh Radhakrishnan :
>>
>>> >>>I have another question is , hdfs mover (A New Data Migration Tool )
>>> know when to move data from hot to cold  automatically ?
>>> While running the tool, it reads the argument and get the separated list
>>> of hdfs files/dirs to migrate. Then it periodically scans these files in
>>> HDFS to check if the block placement satisfies the storage policy, if not
>>> satisfied it moves the replicas to a different storage type in order to
>>> fulfill the storage policy requirement. This cycle continues until it hits
>>> an error or no blocks to move etc. Could you please tell me, what do you
>>> meant by "automatically" ? FYI, HDFS-10285 is proposing an idea to
>>> introduce a daemon thread in Namenode to track the storage movements set by
>>> APIs from clients. This Daemon thread named as StoragePolicySatisfier(SPS)
>>> serves something similar to ReplicationMonitor. If interested you can read
>>> the https://goo.gl/NA5EY0 proposal/idea and welcome feedback.
>>>
>>> Sleep time between each cycle is, ('dfs.heartbeat.interval' * 2000) +
>>> ('dfs.namenode.replication.interval' * 1000) milliseconds;
>>>
>>> >>>It use algorithm like LRU、LFU ?
>>> It will simply iterating over the lists in the order of files/dirs given
>>> to this tool as an argument. afaik, its just maintains the order mentioned
>>> by the user.
>>>
>>> Regards,
>>> Rakesh
>>>
>>>
>>> On Wed, Jul 20, 2016 at 7:05 AM, kevin  wrote:
>>>
>>>> Thanks a lot Rakesh.
>>>>
>>>> I have another question is , hdfs mover (A New Data Migration Tool )
>>>> know when to move data from hot to cold  automatically ? It
>>>> use algorithm like LRU、LFU ?
>>>>
>>>> 2016-07-19 19:55 GMT+08:00 Rakesh Radhakrishnan :
>>>>
>>>>> >>>>Is that mean I should config dfs.replication with 1 ?  if more
>>>>> than one I should not use *Lazy_Persist*  policies ?
>>>>>
>>>>> The idea of Lazy_Persist policy is, while writing blocks, one replica
>>>>> will be placed in memory first and then it is lazily persisted into DISK.
>>>>> It doesn't means that, you are not allowed to configure dfs.replication >
>>>>> 1. If 'dfs.replication' is configured > 1 then the first replica will be
>>>>> placed in RAM_DISK and all the other replicas (n-1) will be written to the
>>>>> DISK. Here the (n-1) replicas will have the overhead of pipeline
>>>>> replication over the network and the DISK write latency on the write hot
>>>>> path. So you will not get better performance results.
>>>>>
>>>>> IIUC, for getting memory latency benefits, it is recommended to use
>>>>> replication=1. In this way, applications should be able

Re: About Archival Storage

[kevin]

Thanks a lot.

2016-07-20 15:20 GMT+08:00 Zheng, Kai :

> There was some related discussion about this in
> https://issues.apache.org/jira/browse/HDFS-7343 where files and blocks
> temperature will be measured across the cluster and used to determine when
> and how to set storage polices accordingly.
>
>
>
> The effort was suspended somehow because the contributors are working on
> HDFS erasure coding feature. It will be revived soon and feedback are
> welcome!
>
>
>
> Regards,
>
> Kai
>
>
>
> *From:* kevin [mailto:kiss.kevin...@gmail.com]
> *Sent:* Wednesday, July 20, 2016 2:48 PM
> *To:* Rakesh Radhakrishnan 
> *Cc:* user.hadoop 
> *Subject:* Re: About Archival Storage
>
>
>
> Ok,thanks.It means that I need to decide which data is hot and which time
> it is cold, then change it storage policy and tell  'Mover tool'  to move
> it.
>
>
>
> 2016-07-20 14:29 GMT+08:00 Rakesh Radhakrishnan :
>
> Based on storage policy the data from hot storage will be moved to cold
> storage. The storage policy defines the number of replicas to be located on
> each storage type. It is possible to change the storage policy on a
> directory(for example: HOT to COLD) and then invoke 'Mover tool' on that
> directory to make the policy effective. One can set/change the storage
> policy via HDFSCommand, "hdfs storagepolicies -setStoragePolicy -path
>  -policy ". After setting the new policy, you need to run the
> tool, then it identifies the replicas to be moved based on the storage
> policy information, and schedules the movement between source and
> destination data nodes to satisfy the policy. Internally, the tool is
> comparing the 'storage type' of a block in order to fulfill the 'storage
> policy' requirement.
>
>
>
> Probably you can refer
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html
> to know more about storage types, storage policies and hdfs commands. Hope
> this helps.
>
>
>
> Rakesh
>
>
>
> On Wed, Jul 20, 2016 at 10:30 AM, kevin  wrote:
>
> Thanks again. "automatically" what I mean is the hdfs mover knows the hot
> data have come to cold , I don't need to tell it what exactly files/dirs
> need to be move now ?
>
> Of course I should tell it what files/dirs need to monitoring.
>
>
>
> 2016-07-20 12:35 GMT+08:00 Rakesh Radhakrishnan :
>
> >>>I have another question is , hdfs mover (A New Data Migration Tool )
> know when to move data from hot to cold  automatically ?
>
> While running the tool, it reads the argument and get the separated list
> of hdfs files/dirs to migrate. Then it periodically scans these files in
> HDFS to check if the block placement satisfies the storage policy, if not
> satisfied it moves the replicas to a different storage type in order to
> fulfill the storage policy requirement. This cycle continues until it hits
> an error or no blocks to move etc. Could you please tell me, what do you
> meant by "automatically" ? FYI, HDFS-10285 is proposing an idea to
> introduce a daemon thread in Namenode to track the storage movements set by
> APIs from clients. This Daemon thread named as StoragePolicySatisfier(SPS)
> serves something similar to ReplicationMonitor. If interested you can read
> the https://goo.gl/NA5EY0 proposal/idea and welcome feedback.
>
>
>
> Sleep time between each cycle is, ('dfs.heartbeat.interval' * 2000) +
> ('dfs.namenode.replication.interval' * 1000) milliseconds;
>
>
>
> >>>It use algorithm like LRU、LFU ?
>
> It will simply iterating over the lists in the order of files/dirs given
> to this tool as an argument. afaik, its just maintains the order mentioned
> by the user.
>
>
>
> Regards,
>
> Rakesh
>
>
>
>
>
> On Wed, Jul 20, 2016 at 7:05 AM, kevin  wrote:
>
> Thanks a lot Rakesh.
>
>
>
> I have another question is , hdfs mover (A New Data Migration Tool ) know
> when to move data from hot to cold  automatically ? It use algorithm
> like LRU、LFU ?
>
>
>
> 2016-07-19 19:55 GMT+08:00 Rakesh Radhakrishnan :
>
> >>>>Is that mean I should config dfs.replication with 1 ?  if more than
> one I should not use *Lazy_Persist*  policies ?
>
>
>
> The idea of Lazy_Persist policy is, while writing blocks, one replica will
> be placed in memory first and then it is lazily persisted into DISK. It
> doesn't means that, you are not allowed to configure dfs.replication > 1.
> If 'dfs.replication' is configured > 1 then the first replica will be
> placed in RAM_DISK and all the other replicas (n-1) will be

hdfs2.7.3 kerberos can not startup

[kevin]

*hi,all:*
*My environment : Centos7.2 hadoop2.7.3 jdk1.8*
*after I config hdfs with kerberos ,I can't start up with sbin/start-dfs.sh*

*::namenode log as below  *

*STARTUP_MSG:   build = Unknown -r Unknown; compiled by 'root' on
2016-09-18T09:05Z*
*STARTUP_MSG:   java = 1.8.0_102*
*/*
*2016-09-20 00:54:05,822 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: registered UNIX signal
handlers for [TERM, HUP, INT]*
*2016-09-20 00:54:05,825 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode []*
*2016-09-20 00:54:06,078 INFO
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties*
*2016-09-20 00:54:06,149 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).*
*2016-09-20 00:54:06,149 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
started*
*2016-09-20 00:54:06,151 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: fs.defaultFS is
hdfs://dmp1.example.com:9000 *
*2016-09-20 00:54:06,152 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use
dmp1.example.com:9000  to access this
namenode/service.*
*2016-09-20 00:54:06,446 INFO
org.apache.hadoop.security.UserGroupInformation: Login successful for user
hadoop/dmp1.example@example.com  using
keytab file /etc/hadoop/conf/hdfs.keytab*
*2016-09-20 00:54:06,472 INFO org.apache.hadoop.hdfs.DFSUtil: Starting web
server as: HTTP/dmp1.example@example.com *
*2016-09-20 00:54:06,475 INFO org.apache.hadoop.hdfs.DFSUtil: Starting
Web-server for hdfs at: https://dmp1.example.com:50470
*
*2016-09-20 00:54:06,517 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog*
*2016-09-20 00:54:06,533 INFO
org.apache.hadoop.security.authentication.server.AuthenticationFilter:
Unable to initialize FileSignerSecretProvider, falling back to use random
secrets.*
*2016-09-20 00:54:06,542 INFO org.apache.hadoop.http.HttpRequestLog: Http
request log for http.requests.namenode is not defined*
*2016-09-20 00:54:06,546 INFO org.apache.hadoop.http.HttpServer2: Added
global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)*
*2016-09-20 00:54:06,548 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context hdfs*
*2016-09-20 00:54:06,548 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context static*
*2016-09-20 00:54:06,548 INFO org.apache.hadoop.http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context logs*
*2016-09-20 00:54:06,653 INFO org.apache.hadoop.http.HttpServer2: Added
filter 'org.apache.hadoop.hdfs.web.AuthFilter'
(class=org.apache.hadoop.hdfs.web.AuthFilter)*
*2016-09-20 00:54:06,654 INFO org.apache.hadoop.http.HttpServer2:
addJerseyResourcePackage:
packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
pathSpec=/webhdfs/v1/**
*2016-09-20 00:54:06,657 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to getDelegationToken*
*2016-09-20 00:54:06,658 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to renewDelegationToken*
*2016-09-20 00:54:06,658 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to cancelDelegationToken*
*2016-09-20 00:54:06,659 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to fsck*
*2016-09-20 00:54:06,659 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to imagetransfer*
*2016-09-20 00:54:06,665 WARN org.mortbay.log:
java.lang.NullPointerException*
*2016-09-20 00:54:06,665 INFO org.apache.hadoop.http.HttpServer2:
HttpServer.start() threw a non Bind IOException*
*java.io.IOException: !JsseListener: java.lang.NullPointerException*
* at
org.mortbay.jetty.security.SslSocketConnector.newServerSocket(SslSocketConnector.java:516)*
* at
org.apache.hadoop.security.ssl.SslSocketConnectorSecure.newServerSocket(SslSocketConnectorSecure.java:47)*
* at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)*
* at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:914)*
* at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:856)*
* at
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)*
* at
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:753)*
* at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:639)*
* at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:812)*
* at
org.apache.hadoop.hdfs.server.namenode.NameNode

Re: hdfs2.7.3 kerberos can not startup

[kevin]

hadoop/dmp2.example@example.com 
(arcfour-hmac)2 09/19/2016 20:21:03 hadoop/dmp3.example@example.com
 (aes256-cts-hmac-sha1-96)2 09/19/2016
20:21:03 hadoop/dmp3.example@example.com 
(aes128-cts-hmac-sha1-96)2 09/19/2016 20:21:03
hadoop/dmp3.example@example.com 
(des3-cbc-sha1)2 09/19/2016 20:21:03
hadoop/dmp3.example@example.com 
(arcfour-hmac) *

2016-09-20 15:52 GMT+08:00 Rakesh Radhakrishnan :

> >>>>>>Caused by: javax.security.auth.login.LoginException: Unable to
> obtain password from user
>
> Could you please check kerberos principal name is specified correctly in
> "hdfs-site.xml", which is used to authenticate against Kerberos.
>
> If keytab file defined in "hdfs-site.xml" and doesn't exists or wrong
> path, you will see
> this error. So, please verify the path and the keytab filename correctly
> configured.
>
> I hope hadoop discussion thread, https://goo.gl/M6l3vv may help you.
>
>
> >>>>>>>2016-09-20 00:54:06,665 INFO org.apache.hadoop.http.HttpServer2:
> HttpServer.start() threw a non Bind IOException
> java.io.IOException: !JsseListener: java.lang.NullPointerException
>
> This is probably due to some missing configuration.
> Could you please re-check the ssl-server.xml, keystore and truststore
> properties:
>
> ssl.server.keystore.location
> ssl.server.keystore.keypassword
> ssl.client.truststore.location
> ssl.client.truststore.password
>
> Rakesh
>
> On Tue, Sep 20, 2016 at 10:53 AM, kevin  wrote:
>
>> *hi,all:*
>> *My environment : Centos7.2 hadoop2.7.3 jdk1.8*
>> *after I config hdfs with kerberos ,I can't start up with
>> sbin/start-dfs.sh*
>>
>> *::namenode log as below  *
>>
>> *STARTUP_MSG:   build = Unknown -r Unknown; compiled by 'root' on
>> 2016-09-18T09:05Z*
>> *STARTUP_MSG:   java = 1.8.0_102*
>> */*
>> *2016-09-20 00:54:05,822 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: registered UNIX signal
>> handlers for [TERM, HUP, INT]*
>> *2016-09-20 00:54:05,825 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode []*
>> *2016-09-20 00:54:06,078 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties*
>> *2016-09-20 00:54:06,149 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).*
>> *2016-09-20 00:54:06,149 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> started*
>> *2016-09-20 00:54:06,151 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: fs.defaultFS is
>> hdfs://dmp1.example.com:9000 <http://dmp1.example.com:9000>*
>> *2016-09-20 00:54:06,152 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use
>> dmp1.example.com:9000 <http://dmp1.example.com:9000> to access this
>> namenode/service.*
>> *2016-09-20 00:54:06,446 INFO
>> org.apache.hadoop.security.UserGroupInformation: Login successful for user
>> hadoop/dmp1.example@example.com  using
>> keytab file /etc/hadoop/conf/hdfs.keytab*
>> *2016-09-20 00:54:06,472 INFO org.apache.hadoop.hdfs.DFSUtil: Starting
>> web server as: HTTP/dmp1.example@example.com
>> *
>> *2016-09-20 00:54:06,475 INFO org.apache.hadoop.hdfs.DFSUtil: Starting
>> Web-server for hdfs at: https://dmp1.example.com:50470
>> <https://dmp1.example.com:50470>*
>> *2016-09-20 00:54:06,517 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog*
>> *2016-09-20 00:54:06,533 INFO
>> org.apache.hadoop.security.authentication.server.AuthenticationFilter:
>> Unable to initialize FileSignerSecretProvider, falling back to use random
>> secrets.*
>> *2016-09-20 00:54:06,542 INFO org.apache.hadoop.http.HttpRequestLog: Http
>> request log for http.requests.namenode is not defined*
>> *2016-09-20 00:54:06,546 INFO org.apache.hadoop.http.HttpServer2: Added
>> global filter 'safety'
>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)*
>> *2016-09-20 00:54:06,548 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context hdfs*
>> *2016-09-20 00:54:06,548 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
>> context static*
>> *2016-09-20 00:

Re: hdfs2.7.3 kerberos can not startup

[kevin]

Thank you Brahma Reddy Battula.
It's because of my problerm of the hdfs-site config file and https
ca configuration.
now I can startup namenode and I can see the datanodes from the web.
but When I try hdfs dfs -ls /:

*[hadoop@dmp1 hadoop-2.7.3]$ hdfs dfs -ls /*
*16/09/20 07:56:48 WARN ipc.Client: Exception encountered while connecting
to the server : javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]*
*ls: Failed on local exception: java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)]; Host Details : local host is:
"dmp1.example.com/192.168.249.129
<http://dmp1.example.com/192.168.249.129>"; destination host is: "dmp1.*
*example**.com":9000; *

current user is hadoop which startup hdfs , and I have add addprinc hadoop
with commond :
kadmin.local -q "addprinc hadoop"


2016-09-20 17:33 GMT+08:00 Brahma Reddy Battula <
brahmareddy.batt...@huawei.com>:

> Seems to be property problem.. it should be *principal* ( “l” is missed).
>
>
>
> **
>
> *  dfs.secondary.namenode.kerberos.principa*
>
> *  hadoop/_h...@example.com *
>
> **
>
>
>
>
>
> For namenode httpserver start fail, please check rakesh comments..
>
>
>
> This is probably due to some missing configuration.
>
> Could you please re-check the ssl-server.xml, keystore and truststore
> properties:
>
>
>
> ssl.server.keystore.location
>
> ssl.server.keystore.keypassword
>
> ssl.client.truststore.location
>
> ssl.client.truststore.password
>
>
>
>
>
> --Brahma Reddy Battula
>
>
>
> *From:* kevin [mailto:kiss.kevin...@gmail.com]
> *Sent:* 20 September 2016 16:53
> *To:* Rakesh Radhakrishnan
> *Cc:* user.hadoop
> *Subject:* Re: hdfs2.7.3 kerberos can not startup
>
>
>
> thanks, but my issue is name node could  *Login successful,but second
> namenode couldn't. and name node got a HttpServer.start() threw a non Bind
> IOException:*
>
>
>
> hdfs-site.xml:
>
>
>
> **
>
> *dfs.webhdfs.enabled*
>
> *true*
>
> **
>
>
>
> **
>
> *  dfs.block.access.token.enable*
>
> *  true*
>
> **
>
>
>
> **
>
> **
>
> *  dfs.namenode.kerberos.principal*
>
> *  hadoop/_h...@example.com *
>
> **
>
> **
>
> *  dfs.namenode.keytab.file*
>
> *  /etc/hadoop/conf/hdfs.keytab*
>
> **
>
> **
>
> *  dfs.https.port*
>
> *  50470*
>
> **
>
> **
>
> *  dfs.namenode.https-address*
>
> *  dmp1.example.com:50470 <http://dmp1.example.com:50470>*
>
> **
>
> **
>
> *  dfs.namenode.kerberos.internal.spnego.principa*
>
> *  HTTP/_h...@example.com *
>
> **
>
> **
>
> *  dfs.web.authentication.kerberos.keytab*
>
> *  /etc/hadoop/conf/hdfs.keytab*
>
> **
>
> **
>
> *  dfs.http.policy*
>
> *  HTTPS_ONLY*
>
> **
>
> **
>
> *  dfs.https.enable*
>
> *  true*
>
> **
>
>
>
>
>
> **
>
> **
>
> *  dfs.namenode.secondary.http-address*
>
> *  dmp1.example.com:50090 <http://dmp1.example.com:50090>*
>
> **
>
> **
>
> *  dfs.secondary.namenode.keytab.file*
>
> *  /etc/hadoop/conf/hdfs.keytab*
>
> **
>
> **
>
> *  dfs.secondary.namenode.kerberos.principa*
>
> *  hadoop/_h...@example.com *
>
> *  *
>
> **
>
> *  dfs.secondary.namenode.kerberos.internal.spnego.principal*
>
> *  HTTP/_h...@example.com *
>
> **
>
> **
>
> *  dfs.namenode.secondary.https-port*
>
> *  50470*
>
> **
>
>
>
>
>
> **
>
>
>
> **
>
> *  dfs.journalnode.keytab.file*
>
> *  /etc/hadoop/conf/hdfs.keytab*
>
> **
>
> **
>
> *  dfs.journalnode.kerberos.principa*
>
> *  hadoop/_h...@example.com *
>
> *  *
>
> **
>
> *  dfs.journalnode.kerberos.internal.spnego.principa*
>
> *  HTTP/_h...@example.com *
>
> **
>
> **
>
> *  dfs.web.authentication.kerberos.keytab*
>
> *  /etc/hadoop/conf/hdfs.keytab*
>
> **
>
>
>
>
>
> **
>
> **
>
> *  dfs.datanode.kerberos.principal*
>
> *  hadoop/_h...@example.com *
>
> **
>
> **
>
> *  dfs.datanode.keytab.file*
>
> *  /etc/hadoop/conf/hdfs.keytab*
>
> **
>
> **
>
> *  dfs.datanode.data.dir.perm*
>
> *  700*
>
> **
>
>
>
> **
>
> **
>
> *  dfs.datanode.address*
>
> *  0.0.0.0:61004 <http://0.0.0.0:610

Re: hdfs2.7.3 kerberos can not startup

[kevin]

[hadoop@dmp1 ~]$ hdfs dfs -ls /
16/09/20 15:00:44 WARN ipc.Client: Exception encountered while connecting
to the server : javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)]; Host Details : local host is: "
dmp1.youedata.com/192.168.249.129"; destination host is: "dmp1.youedata.com
":9000;
[hadoop@dmp1 ~]$ klist
Ticket cache: KEYRING:persistent:1004:1004
Default principal: had...@example.com

Valid starting   Expires  Service principal
09/20/2016 14:57:34  09/21/2016 14:57:31  krbtgt/example@example.com
renew until 09/27/2016 14:57:31
[hadoop@dmp1 ~]$

I have run kinit had...@example.com before .

2016-09-21 10:14 GMT+08:00 Wei-Chiu Chuang :

> You need to run kinit command to authenticate before running hdfs dfs -ls
> command.
>
> Wei-Chiu Chuang
>
> On Sep 20, 2016, at 6:59 PM, kevin  wrote:
>
> Thank you Brahma Reddy Battula.
> It's because of my problerm of the hdfs-site config file and https
> ca configuration.
> now I can startup namenode and I can see the datanodes from the web.
> but When I try hdfs dfs -ls /:
>
> *[hadoop@dmp1 hadoop-2.7.3]$ hdfs dfs -ls /*
> *16/09/20 07:56:48 WARN ipc.Client: Exception encountered while connecting
> to the server : javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)]*
> *ls: Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]; Host Details : local host is:
> "dmp1.example.com/192.168.249.129
> <http://dmp1.example.com/192.168.249.129>"; destination host is: "dmp1.*
> *example**.com":9000; *
>
> current user is hadoop which startup hdfs , and I have add addprinc hadoop
> with commond :
> kadmin.local -q "addprinc hadoop"
>
>
> 2016-09-20 17:33 GMT+08:00 Brahma Reddy Battula <
> brahmareddy.batt...@huawei.com>:
>
>> Seems to be property problem.. it should be *principal* ( “l” is missed).
>>
>>
>>
>> **
>>
>> *  dfs.secondary.namenode.kerberos.principa*
>>
>> *  hadoop/_h...@example.com *
>>
>> **
>>
>>
>>
>>
>>
>> For namenode httpserver start fail, please check rakesh comments..
>>
>>
>>
>> This is probably due to some missing configuration.
>>
>> Could you please re-check the ssl-server.xml, keystore and truststore
>> properties:
>>
>>
>>
>> ssl.server.keystore.location
>>
>> ssl.server.keystore.keypassword
>>
>> ssl.client.truststore.location
>>
>> ssl.client.truststore.password
>>
>>
>>
>>
>>
>> --Brahma Reddy Battula
>>
>>
>>
>> *From:* kevin [mailto:kiss.kevin...@gmail.com]
>> *Sent:* 20 September 2016 16:53
>> *To:* Rakesh Radhakrishnan
>> *Cc:* user.hadoop
>> *Subject:* Re: hdfs2.7.3 kerberos can not startup
>>
>>
>>
>> thanks, but my issue is name node could  *Login successful,but second
>> namenode couldn't. and name node got a HttpServer.start() threw a non Bind
>> IOException:*
>>
>>
>>
>> hdfs-site.xml:
>>
>>
>>
>> **
>>
>> *dfs.webhdfs.enabled*
>>
>> *true*
>>
>> **
>>
>>
>>
>> **
>>
>> *  dfs.block.access.token.enable*
>>
>> *  true*
>>
>> **
>>
>>
>>
>> **
>>
>> **
>>
>> *  dfs.namenode.kerberos.pr
>> <http://dfs.namenode.kerberos.pr>incipal*
>>
>> *  hadoop/_h...@example.com *
>>
>> **
>>
>> **
>>
>> *  dfs.namenode.keytab.file*
>>
>> *  /etc/hadoop/conf/hdfs.keytab*
>>
>> **
>>
>> **
>>
>> *  dfs.https.port*
>>
>> *  50470*
>>
>> **
>>
>> **
>>
>> *  dfs.namenode.https-address*
>>
>> *  dmp1.example.com:50470 <http://dmp1.example.com:50470/>*
>>
>> **
>>
>> **
>>
>> *  dfs.namenode.kerberos.in
>> <http://dfs.namenode.kerberos.in>ternal.spnego.principa*
>>
>> *  HTTP/_h...@exa

Import with Sqoop

[Kevin Burton]

I execute the line:

 

sqoop import --connect
'jdbc:sqlserver://nbreports:1433;databaseName=productcatalog' --username
 --password  --table CatalogProducts

 

And I get the following output:

 

Warning: /usr/lib/hbase does not exist! HBase imports will fail.

Please set $HBASE_HOME to the root of your HBase installation.

13/04/23 18:00:36 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead.

13/04/23 18:00:36 INFO manager.SqlManager: Using default fetchSize of 1000

. . . . .



 

But it doesn't seem to do anything. I executed 'hadoop fs -ls' and I didn't
see anything. Any ideas what I have done wrong?

 

Kevin

Maven dependency

[Kevin Burton]

I am reading "Hadoop in Action" and the author on page 51 puts forth this
code:

 

public class WordCount2 { 

public static void main(String[] args) { 

   JobClient client = new JobClient(); 

   JobConf conf = new JobConf(WordCount2.class); 

   FileInputFormat.addInputPath(conf, new Path(args[0]));

   FileOutputFormat.setOutputPath(conf, new Path(args[1])); 

   conf.setOutputKeyClass(Text.class);

   conf.setOutputValueClass(LongWritable.class);

   conf.setMapperClass(TokenCountMapper.class);

   conf.setCombinerClass(LongSumReducer.class);

   conf.setReducerClass(LongSumReducer.class);r

   client.setConf(conf);

   try {

   JobClient.runJob(conf);

   } catch (Exception e) {

   e.printStackTrace();

   } 

   } 

}

 

Which is an example for a simple MapReduce job. But being a beginner I am
not sure how to set up a project for this code. If I am using Maven what are
the Maven dependencies that I need? There are several map reduce
dependencies and I am not sure which to pick. Are there other dependencies
need (such as JobConf)? What are the imports needed? During the construction
of the configuration what heuristics are used to find the configuration for
the Hadoop cluster?

 

Thank you.

Comparison between JobClient/JobConf and Job/Configuration

[Kevin Burton]

I notice that in some beginning texts on starting a Hadoop MapReduce job
sometimes JobClient/JobConf is used and sometimes Job/Configuration is used.
I have yet to see anyone comment on the features/benefits of either set of
methods. Could someone comment on their preferred method for starting a
MapReduce job from a Java program? 

 

Thank you.

Re: M/R job to a cluster?

[Kevin Burton]

It is hdfs://devubuntu05:9000. Is this wrong? Devubuntu05 is the name of the 
host where the NameNode and JobTracker should be running. It is also the host 
where I am running the M/R client code.

On Apr 26, 2013, at 4:06 PM, Rishi Yadav  wrote:

> check core-site.xml and see value of fs.default.name. if it has localhost you 
> are running locally.
> 
> 
> 
> 
> On Fri, Apr 26, 2013 at 1:59 PM,  wrote:
>> I suspect that my MapReduce job is being run locally. I don't have any 
>> evidence but I am not sure how the specifics of my configuration are 
>> communicated to the Java code that I write. Based on the text that I have 
>> read online basically I start with code like:
>> 
>> JobClient client = new JobClient();
>> JobConf conf - new JobConf(WordCount.class);
>> . . . . .
>> 
>> Where do I communicate the configuration information so that the M/R job 
>> runs on the cluster and not locally? Or is the configuration location 
>> "magically determined"?
>> 
>> Thank you.
> 

RE: M/R Staticstics

[Kevin Burton]

Answers below.

 

From: Omkar Joshi [mailto:ojo...@hortonworks.com] 
Sent: Friday, April 26, 2013 7:15 PM
To: user@hadoop.apache.org
Subject: Re: M/R Staticstics

 

Have you enabled security?

No

 

can you share the output for your hdfs?

 

bin/hadoop fs -ls /

 

kevin@devUbuntu05:~$ hadoop fs -ls /

Found 2 items

drwxrwxrwx   - hdfs supergroup  0 2013-04-26 13:33 /tmp

drwxr-xr-x   - hdfs supergroup  0 2013-04-19 16:40 /user

 

and is /tmp/hadoop-yarn/staging/history/done directory present in hdfs ? if
so then what permissions?

 

kevin@devUbuntu05:~$ hadoop fs -ls -R /tmp

drwxrwx---   - mapred supergroup  0 2013-04-26 13:33
/tmp/hadoop-yarn

ls: Permission denied: user=kevin, access=READ_EXECUTE,
inode="/tmp/hadoop-yarn":mapred:supergroup:drwxrwx---

 

 

kevin@devUbuntu05:~$ sudo -u hdfs hadoop fs -ls -R /tmp

[sudo] password for kevin:

drwxrwx---   - mapred supergroup  0 2013-04-26 13:33
/tmp/hadoop-yarn

drwxrwx---   - mapred supergroup  0 2013-04-26 13:33
/tmp/hadoop-yarn/staging

drwxrwx---   - mapred supergroup  0 2013-04-26 13:33
/tmp/hadoop-yarn/staging/history

drwxrwx---   - mapred supergroup  0 2013-04-26 13:33
/tmp/hadoop-yarn/staging/history/done

drwxrwxrwt   - mapred supergroup  0 2013-04-26 13:33
/tmp/hadoop-yarn/staging/history/done_intermediate

kevin@devUbuntu05:~$

 

also please share exception stack trace...

 

There is no exception now that I created /tmp on HDFS. But I still cannot
see the logs via port 50030 on the master. In other words nothing seems to
be listening on http:devubuntu05:50030. The log for map reduce looks like:

 

2013-04-26 13:35:26,107 INFO
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService: Instantiated
MRClientService at devUbuntu05/172.16.26.68:10020

2013-04-26 13:35:26,107 INFO org.apache.hadoop.yarn.service.AbstractService:
Service:HistoryClientService is started.

2013-04-26 13:35:26,107 INFO org.apache.hadoop.yarn.service.AbstractService:
Service:org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer is started.

2013-04-26 13:35:55,290 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
History Cleaner started

2013-04-26 13:35:55,295 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
History Cleaner complete

2013-04-26 13:38:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
Starting scan to move intermediate done files

2013-04-26 13:41:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
Starting scan to move intermediate done files

2013-04-26 13:44:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
Starting scan to move intermediate done files

2013-04-26 13:47:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
Starting scan to move intermediate done files

2013-04-26 13:50:25,283 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
Starting scan to move intermediate done files

 

 

Thanks,

Omkar Joshi

Hortonworks Inc

 

On Fri, Apr 26, 2013 at 3:05 PM,  wrote:

  

I was able to overcome the permission exception in the log by creating an
HDFS tmp folder (hadoop fs -mkdir /tmp) and opening it up to the world
(hadoop fs -chmod a+rwx /tmp). That got rid of the exception put I still am
able to connect to port 50030 to see M/R status. More ideas?

 

Even though the exception was missing from the logs of one server in the
cluster, l looked on another server and found essentially the same
permission problem:

 

2013-04-26 13:34:56,462 FATAL
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: Error starting
JobHistoryServer

org.apache.hadoop.yarn.YarnException: Error creating done directory:
[hdfs://devubuntu05:9000/tmp/hadoop-yarn/staging/history/done]

at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.init(HistoryFileManager
.java:424)

at
org.apache.hadoop.mapreduce.v2.hs.JobHistory.init(JobHistory.java:87)

at
org.apache.hadoop.yarn.service.CompositeService.init(CompositeService.java:5
8)

 

. . . . .

 

On Fri, Apr 26, 2013 at 10:37 AM, Rishi Yadav wrote: 

 

  do you see "retired jobs" on job tracker page. There is also "job tracker
history" on the bottom of page.  

 

something like this  http://nn.zettabyte.com:50030/jobtracker.jsp

Thanks and Regards, 

Rishi Yadav 







On Fri, Apr 26, 2013 at 7:36 AM, < rkevinbur...@charter.net> wrote: 

When I submit a simple "Hello World" M/R job like WordCount it takes less
than 5 seconds. The texts show numerous methods for monitoring M/R jobs as
they are happening but I have yet to see any that show statistics about a
job after it has completed. Obviously simple jobs that take a short amount
of time don't allow time to fire up any web mage or monitoring tool to see
how it progresses through the JobTracker and TaskTracker as well as which
node it is processed on. Any suggestions on how could see this kind of data
*after* a job has completed? 

 

 

Re: Warnings?

[Kevin Burton]

Is the native library not available for Ubuntu? If so how do I load it?

Can I tell which key is off? Since I am just starting I would want to be as up 
to date as possible. It is out of date probably because I copied my examples 
from books and tutorials.

The main class does derive from Tool. Should I ignore this warning as it seems 
to be in error?

Thank you.

On Apr 26, 2013, at 7:49 PM, Ted Xu  wrote:

> Hi,
> 
> First warning is saying hadoop cannot load native library, usually a 
> compression codec. In that case, hadoop will use java implementation instead, 
> which is slower.
> 
> Second is caused by hadoop 1.x/2.x configuration key change. You're using a 
> 1.x style key under 2.x, yet hadoop still guarantees backward compatibility.
> 
> Third is saying that the main class of a hadoop application is recommanded to 
> implement org.apache.hadoop.util.Tool, or else generic command line options 
> (e.g., -D options) will not supported.   
> 
> 
> On Sat, Apr 27, 2013 at 5:51 AM,  wrote:
>> I am running a simple WordCount m/r job and I get output but I get five 
>> warnings that I am not sure if I should pay attention to:
>> 
>> 13/04/26 16:24:50 WARN util.NativeCodeLoader: Unable to load native-hadoop 
>> library for your platform... using builtin-java classes where applicable
>> 
>> 13/04/26 16:24:50 WARN conf.Configuration: session.id is deprecated. 
>> Instead, use dfs.metrics.session-id
>> 
>> 13/04/26 16:24:50 WARN mapred.JobClient: Use GenericOptionsParser for 
>> parsing the arguments. Applications should implement Tool for the same.
>> 
>> 13/04/26 16:24:51 WARN mapreduce.Counters: Group 
>> org.apache.hadoop.mapred.Task$Counter is deprecated. Use 
>> org.apache.hadoop.mapreduce.TaskCounter instead
>> 
>> 13/04/26 16:24:51 WARN mapreduce.Counters: Counter name MAP_INPUT_BYTES is 
>> deprecated. Use FileInputFormatCounters as group name and  BYTES_READ as 
>> counter name instead
>> 
>> Any ideas on what these mean? The only one that I can see in the code is the 
>> third one. I am using GenericOptionsParser as it is part of an example that 
>> I copied. But I don't know why this is considered bad.
>> 
>> Thank you.
> 
> 
> 
> -- 
> Regards,
> Ted Xu

Re: M/R job to a cluster?

[Kevin Burton]

Part of the problem is nothing comes up on port 50030. 50070 yes but 50030 no. 

On Apr 28, 2013, at 12:04 PM, shashwat shriparv  
wrote:

> check in namenode:50030 if it appears there its not running in localmode else 
> it is
> 
> Thanks & Regards   
> ∞
> Shashwat Shriparv
> 
> 
> 
> On Sun, Apr 28, 2013 at 1:18 AM, sudhakara st  wrote:
>> Hello Kevin,
>> 
>> In the case:
>> 
>> JobClient client = new JobClient();
>> JobConf conf - new JobConf(WordCount.class);
>> 
>> Job client(default in local system) picks  configuration information  by 
>> referring HADOOP_HOME in local system.
>> 
>> if your job configuration like this:
>> Configuration conf = new Configuration();
>> conf.set("fs.default.name", "hdfs://name_node:9000");
>> conf.set("mapred.job.tracker", "job_tracker_node:9001");
>> 
>> It pickups configuration information  by referring HADOOP_HOME in specified 
>> namenode and job tracker.
>> 
>> Regards,
>> Sudhakara.st
>> 
>> 
>> On Sat, Apr 27, 2013 at 2:52 AM, Kevin Burton  
>> wrote:
>>> It is hdfs://devubuntu05:9000. Is this wrong? Devubuntu05 is the name of 
>>> the host where the NameNode and JobTracker should be running. It is also 
>>> the host where I am running the M/R client code.
>>> 
>>> On Apr 26, 2013, at 4:06 PM, Rishi Yadav  wrote:
>>> 
>>>> check core-site.xml and see value of fs.default.name. if it has localhost 
>>>> you are running locally.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Fri, Apr 26, 2013 at 1:59 PM,   wrote:
>>>>> I suspect that my MapReduce job is being run locally. I don't have any 
>>>>> evidence but I am not sure how the specifics of my configuration are 
>>>>> communicated to the Java code that I write. Based on the text that I have 
>>>>> read online basically I start with code like:
>>>>> 
>>>>> JobClient client = new JobClient();
>>>>> JobConf conf - new JobConf(WordCount.class);
>>>>> . . . . .
>>>>> 
>>>>> Where do I communicate the configuration information so that the M/R job 
>>>>> runs on the cluster and not locally? Or is the configuration location 
>>>>> "magically determined"?
>>>>> 
>>>>> Thank you.
>> 
>> 
>> 
>> -- 
>>
>> Regards,
>> .  Sudhakara.st
> 

RE: Warnings?

[Kevin Burton]

I looked at the link you provided and found the Ubuntu is one of the
"supported platforms" but it doesn't give any information on how to obtain
it or build it. Any idea why it is not includde as part of the Cloudera CDH4
distribution? I followed the installation instructions (mostly apt-get
install . . . .) but I fail to see the libhadoop.so.  In order to avoid this
warning do I need to download the Apache distribution? Which one?

 

For the warnings about the configuration I looked in my configuration and
for this specific example I don't see 'session.id' used anywhere. It must be
used by default. If so why is the deprecated default being used? 

 

As for the two warnings about counters. I know I have not implemented any
code for counters so again this must be something internal. Is there
something I am doing to trigger this?

 

So I can avoid them what are "hadoop generic options"?

 

Thanks again.

 

Kevin

 

From: Ted Xu [mailto:t...@gopivotal.com] 
Sent: Friday, April 26, 2013 10:49 PM
To: user@hadoop.apache.org
Subject: Re: Warnings?

 

Hi Kevin,

 

Please see my comments inline,

 

On Sat, Apr 27, 2013 at 11:24 AM, Kevin Burton 
wrote:

Is the native library not available for Ubuntu? If so how do I load it?

Native libraries usually requires recompile, for more information please
refer Native Libraries
<http://hadoop.apache.org/docs/r2.0.4-alpha/hadoop-project-dist/hadoop-commo
n/NativeLibraries.html> . 

 

 

Can I tell which key is off? Since I am just starting I would want to be as
up to date as possible. It is out of date probably because I copied my
examples from books and tutorials.

 

I think the warning messages are telling it already, "xxx is deprecated, use
xxx instead...". In fact, most of the configure keys are changed from hadoop
1.x to 2.x. The compatibility change may later documented on
http://wiki.apache.org/hadoop/Compatibility.

 

The main class does derive from Tool. Should I ignore this warning as it
seems to be in error?

Of course you can ignore this warning as long as you don't use hadoop
generic options.

 

 

Thank you.


On Apr 26, 2013, at 7:49 PM, Ted Xu  wrote:

Hi,

 

First warning is saying hadoop cannot load native library, usually a
compression codec. In that case, hadoop will use java implementation
instead, which is slower.

 

Second is caused by hadoop 1.x/2.x configuration key change. You're using a
1.x style key under 2.x, yet hadoop still guarantees backward compatibility.

 

Third is saying that the main class of a hadoop application is recommanded
to implement org.apache.hadoop.util.Tool
<http://hadoop.apache.org/docs/current/api/org/apache/hadoop/util/Tool.html>
, or else generic command line options (e.g., -D options) will not
supported.   

 

On Sat, Apr 27, 2013 at 5:51 AM,  wrote:

I am running a simple WordCount m/r job and I get output but I get five
warnings that I am not sure if I should pay attention to: 

 

13/04/26 16:24:50 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable 

 

13/04/26 16:24:50 WARN conf.Configuration: session.id is deprecated.
Instead, use dfs.metrics.session-id 

 

13/04/26 16:24:50 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same. 

 

13/04/26 16:24:51 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.Task$Counter is deprecated. Use
org.apache.hadoop.mapreduce.TaskCounter instead 

 

13/04/26 16:24:51 WARN mapreduce.Counters: Counter name MAP_INPUT_BYTES is
deprecated. Use FileInputFormatCounters as group name and  BYTES_READ as
counter name instead

 

Any ideas on what these mean? The only one that I can see in the code is the
third one. I am using GenericOptionsParser as it is part of an example that
I copied. But I don't know why this is considered bad. 

 

Thank you. 

 





 

-- 

Regards,

Ted Xu





 

-- 

Regards,

Ted Xu

Re: Warnings?

[Kevin Burton]

If it doesn't work what are my options? Is there source that I can download and 
compile?

On Apr 29, 2013, at 10:31 AM, Ted Xu  wrote:

> Hi Kevin,
> 
> Native libraries are those implemented using C/C++, which only provide code 
> level portability (instead of binary level portability, as Java do). That is 
> to say, the binaries provided by CDH4 distribution will in most cases be 
> broken in your environment. 
> 
> To check if your native libraries are working or not, you can follow the 
> instructions I sent previously. Quote as following.
> 
> 
> During runtime, check the hadoop log files for your MapReduce tasks.
> 
> If everything is all right, then: DEBUG util.NativeCodeLoader - Trying to 
> load the custom-built native-hadoop library... INFO util.NativeCodeLoader - 
> Loaded the native-hadoop library
> If something goes wrong, then: INFO util.NativeCodeLoader - Unable to load 
> native-hadoop library for your platform... using builtin-java classes where 
> applicable
> 
> 
> 
> On Mon, Apr 29, 2013 at 10:21 AM, Kevin Burton  
> wrote:
>> I looked at the link you provided and found the Ubuntu is one of the 
>> “supported platforms” but it doesn’t give any information on how to obtain 
>> it or build it. Any idea why it is not includde as part of the Cloudera CDH4 
>> distribution? I followed the installation instructions (mostly apt-get 
>> install . . . .) but I fail to see the libhadoop.so.  In order to avoid this 
>> warning do I need to download the Apache distribution? Which one?
>> 
>>  
>> 
>> For the warnings about the configuration I looked in my configuration and 
>> for this specific example I don’t see ‘session.id’ used anywhere. It must be 
>> used by default. If so why is the deprecated default being used?
>> 
>>  
>> 
>> As for the two warnings about counters. I know I have not implemented any 
>> code for counters so again this must be something internal. Is there 
>> something I am doing to trigger this?
>> 
>>  
>> 
>> So I can avoid them what are “hadoop generic options”?
>> 
>>  
>> 
>> Thanks again.
>> 
>>  
>> 
>> Kevin
>> 
>>  
>> 
>> From: Ted Xu [mailto:t...@gopivotal.com] 
>> Sent: Friday, April 26, 2013 10:49 PM
>> To: user@hadoop.apache.org
>> Subject: Re: Warnings?
>> 
>>  
>> 
>> Hi Kevin,
>> 
>>  
>> 
>> Please see my comments inline,
>> 
>>  
>> 
>> On Sat, Apr 27, 2013 at 11:24 AM, Kevin Burton  
>> wrote:
>> 
>> Is the native library not available for Ubuntu? If so how do I load it?
>> 
>> Native libraries usually requires recompile, for more information please 
>> refer Native Libraries. 
>> 
>>  
>> 
>>  
>> 
>> Can I tell which key is off? Since I am just starting I would want to be as 
>> up to date as possible. It is out of date probably because I copied my 
>> examples from books and tutorials.
>> 
>>  
>> 
>> I think the warning messages are telling it already, "xxx is deprecated, use 
>> xxx instead...". In fact, most of the configure keys are changed from hadoop 
>> 1.x to 2.x. The compatibility change may later documented on 
>> http://wiki.apache.org/hadoop/Compatibility.
>> 
>>  
>> 
>> The main class does derive from Tool. Should I ignore this warning as it 
>> seems to be in error?
>> 
>> Of course you can ignore this warning as long as you don't use hadoop 
>> generic options.
>> 
>>  
>> 
>>  
>> 
>> Thank you.
>> 
>> 
>> On Apr 26, 2013, at 7:49 PM, Ted Xu  wrote:
>> 
>> Hi,
>> 
>>  
>> 
>> First warning is saying hadoop cannot load native library, usually a 
>> compression codec. In that case, hadoop will use java implementation 
>> instead, which is slower.
>> 
>>  
>> 
>> Second is caused by hadoop 1.x/2.x configuration key change. You're using a 
>> 1.x style key under 2.x, yet hadoop still guarantees backward compatibility.
>> 
>>  
>> 
>> Third is saying that the main class of a hadoop application is recommanded 
>> to implement org.apache.hadoop.util.Tool, or else generic command line 
>> options (e.g., -D options) will not supported.   
>> 
>>  
>> 
>> On Sat, Apr 27, 2013 at 5:51 AM,  wrote:
>> 
>> I am running a simple WordCount m/r job and I get output but I get five 
>> warnings that I am not sure if I should pay attention to:
>> 
>>  
>> 
>> 13

Re: Incompartible cluserIDS

[Kevin Burton]

Thank you the HDFS system seems to be up. Now I am having a problem with 
getting the JobTracker and TaskTracker up. According to the logs on the 
JobTracker mapred doesn't have  write permission to /. I am not clear on what 
the permissions should be.

Anyway, thank you.

On Apr 29, 2013, at 4:30 PM, Mohammad Tariq  wrote:

> Hello Kevin,
> 
>   Have you reformatted the NN(unsuccessfully)?Was your NN serving 
> some other cluster earlier or your DNs were part of some other 
> cluster?Datanodes bind themselves to namenode through namespaceID and in your 
> case the IDs of DNs and NN seem to be different. As a workaround you could do 
> this :
> 
> 1- Stop all the daemons.
> 2- Go to the directory which you have specified as the value of 
> "dfs.name.dir" property in your hdfs-site.xml file.
> 3- You'll find a directory called "current" inside this directory where a 
> file named "VERSION" will be present. Open this file and copy the value of 
> "namespaceID" form here.
> 4- Now go to the directory which you have specified as the value of 
> "dfs.data.dir" property in your hdfs-site.xml file.
> 5- Move inside the "current" directory and open the "VERSION" file here as 
> well. Now replace the value of "namespaceID" present here with the one you 
> had copied earlier.
> 6- Restart all the daemons.
> 
> Note : If you have not created dfs.name.dir and dfs.data.dir separately, you 
> could find all this inside your temp directory.
> 
> HTH
> 
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
> 
> 
> On Tue, Apr 30, 2013 at 2:45 AM,  wrote:
>> I am trying to start up a cluster and in the datanode log on the NameNode 
>> server I get the error:
>> 
>> 2013-04-29 15:50:20,988 INFO org.apache.hadoop.hdfs.server.common.Storage: 
>> Lock on /data/hadoop/dfs/data/in_use.lock acquired by nodename 
>> 1406@devUbuntu05
>> 2013-04-29 15:50:20,990 FATAL 
>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for 
>> block pool Block pool BP-1306349046-172.16.26.68-1367256199559 (storage id 
>> DS-403514403-172.16.26.68-50010-1366406077018) service to 
>> devUbuntu05/172.16.26.68:9000
>> java.io.IOException: Incompatible clusterIDs in /data/hadoop/dfs/data: 
>> namenode clusterID = CID-23b9f9c7-2c25-411f-8bd2-4d5c9d7c25a1; datanode 
>> clusterID = CID-e3f6b811-c1b4-4778-a31e-14dea8b2cca8
>> at 
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:391)
>> at 
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:191)
>> at 
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:219)
>> 
>> How do I get around this error? What does the error mean?
>> 
>> Thank you.
>> 
>> Kevin
> 

Re: Incompartible cluserIDS

[Kevin Burton]

"It" is '/'?

On Apr 29, 2013, at 5:09 PM, Mohammad Tariq  wrote:

> make it 755.
> 
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
> 
> 
> On Tue, Apr 30, 2013 at 3:30 AM, Kevin Burton  
> wrote:
>> Thank you the HDFS system seems to be up. Now I am having a problem with 
>> getting the JobTracker and TaskTracker up. According to the logs on the 
>> JobTracker mapred doesn't have  write permission to /. I am not clear on 
>> what the permissions should be.
>> 
>> Anyway, thank you.
>> 
>> On Apr 29, 2013, at 4:30 PM, Mohammad Tariq  wrote:
>> 
>>> Hello Kevin,
>>> 
>>>   Have you reformatted the NN(unsuccessfully)?Was your NN serving 
>>> some other cluster earlier or your DNs were part of some other 
>>> cluster?Datanodes bind themselves to namenode through namespaceID and in 
>>> your case the IDs of DNs and NN seem to be different. As a workaround you 
>>> could do this :
>>> 
>>> 1- Stop all the daemons.
>>> 2- Go to the directory which you have specified as the value of 
>>> "dfs.name.dir" property in your hdfs-site.xml file.
>>> 3- You'll find a directory called "current" inside this directory where a 
>>> file named "VERSION" will be present. Open this file and copy the value of 
>>> "namespaceID" form here.
>>> 4- Now go to the directory which you have specified as the value of 
>>> "dfs.data.dir" property in your hdfs-site.xml file.
>>> 5- Move inside the "current" directory and open the "VERSION" file here as 
>>> well. Now replace the value of "namespaceID" present here with the one you 
>>> had copied earlier.
>>> 6- Restart all the daemons.
>>> 
>>> Note : If you have not created dfs.name.dir and dfs.data.dir separately, 
>>> you could find all this inside your temp directory.
>>> 
>>> HTH
>>> 
>>> Warm Regards,
>>> Tariq
>>> https://mtariq.jux.com/
>>> cloudfront.blogspot.com
>>> 
>>> 
>>> On Tue, Apr 30, 2013 at 2:45 AM,   wrote:
>>>> I am trying to start up a cluster and in the datanode log on the NameNode 
>>>> server I get the error:
>>>> 
>>>> 2013-04-29 15:50:20,988 INFO org.apache.hadoop.hdfs.server.common.Storage: 
>>>> Lock on /data/hadoop/dfs/data/in_use.lock acquired by nodename 
>>>> 1406@devUbuntu05
>>>> 2013-04-29 15:50:20,990 FATAL 
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for 
>>>> block pool Block pool BP-1306349046-172.16.26.68-1367256199559 (storage id 
>>>> DS-403514403-172.16.26.68-50010-1366406077018) service to 
>>>> devUbuntu05/172.16.26.68:9000
>>>> java.io.IOException: Incompatible clusterIDs in /data/hadoop/dfs/data: 
>>>> namenode clusterID = CID-23b9f9c7-2c25-411f-8bd2-4d5c9d7c25a1; datanode 
>>>> clusterID = CID-e3f6b811-c1b4-4778-a31e-14dea8b2cca8
>>>> at 
>>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:391)
>>>> at 
>>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:191)
>>>> at 
>>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:219)
>>>> 
>>>> How do I get around this error? What does the error mean?
>>>> 
>>>> Thank you.
>>>> 
>>>> Kevin
> 

RE: Permission problem

[Kevin Burton]

I have relaxed it even further so now it is 775

 

kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$ hadoop fs -ls -d /

Found 1 items

drwxrwxr-x   - hdfs supergroup  0 2013-04-29 15:43 /

 

But I still get this error:

 

2013-04-30 07:43:02,520 FATAL org.apache.hadoop.mapred.JobTracker:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

 

 

From: Mohammad Tariq [mailto:donta...@gmail.com] 
Sent: Monday, April 29, 2013 5:10 PM
To: user@hadoop.apache.org
Subject: Re: Incompartible cluserIDS

 

make it 755.




Warm Regards,

Tariq

https://mtariq.jux.com/

cloudfront.blogspot.com

 

 

RE: Permission problem

[Kevin Burton]

That is what I perceive as the problem. The hdfs file system was created
with the user 'hdfs' owning the root ('/') but for some reason with a M/R
job the user 'mapred' needs to have write permission to the root. I don't
know how to satisfy both conditions. That is one reason that I relaxed the
permission to 775 so that the group would also have write permission but
that didn't seem to help.

 

From: Mohammad Tariq [mailto:donta...@gmail.com] 
Sent: Tuesday, April 30, 2013 8:20 AM
To: Kevin Burton
Subject: Re: Permission problem

 

user?"ls" shows "hdfs" and the log says "mapred"..




Warm Regards,

Tariq

https://mtariq.jux.com/

cloudfront.blogspot.com

 

On Tue, Apr 30, 2013 at 6:22 PM, Kevin Burton 
wrote:

I have relaxed it even further so now it is 775

 

kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$ hadoop fs -ls -d /

Found 1 items

drwxrwxr-x   - hdfs supergroup  0 2013-04-29 15:43 /

 

But I still get this error:

 

2013-04-30 07:43:02,520 FATAL org.apache.hadoop.mapred.JobTracker:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

 

 

From: Mohammad Tariq [mailto:donta...@gmail.com] 
Sent: Monday, April 29, 2013 5:10 PM
To: user@hadoop.apache.org
Subject: Re: Incompartible cluserIDS

 

make it 755.




Warm Regards,

Tariq

https://mtariq.jux.com/

cloudfront.blogspot.com

 

 

 

RE: Permission problem

[Kevin Burton]

To further complicate the issue the log file in
(/var/log/hadoop-0.20-mapreduce/hadoop-hadoop-jobtracker-devUbuntu05.log) is
owned by mapred:mapred and the name of the file seems to indicate some other
lineage (hadoop,hadoop). I am out of my league in understanding the
permission structure for hadoop hdfs and mr. Ideas?

 

From: Kevin Burton [mailto:rkevinbur...@charter.net] 
Sent: Tuesday, April 30, 2013 8:31 AM
To: user@hadoop.apache.org
Cc: 'Mohammad Tariq'
Subject: RE: Permission problem

 

That is what I perceive as the problem. The hdfs file system was created
with the user 'hdfs' owning the root ('/') but for some reason with a M/R
job the user 'mapred' needs to have write permission to the root. I don't
know how to satisfy both conditions. That is one reason that I relaxed the
permission to 775 so that the group would also have write permission but
that didn't seem to help.

 

From: Mohammad Tariq [mailto:donta...@gmail.com] 
Sent: Tuesday, April 30, 2013 8:20 AM
To: Kevin Burton
Subject: Re: Permission problem

 

user?"ls" shows "hdfs" and the log says "mapred"..




Warm Regards,

Tariq

https://mtariq.jux.com/

cloudfront.blogspot.com

 

On Tue, Apr 30, 2013 at 6:22 PM, Kevin Burton 
wrote:

I have relaxed it even further so now it is 775

 

kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$ hadoop fs -ls -d /

Found 1 items

drwxrwxr-x   - hdfs supergroup  0 2013-04-29 15:43 /

 

But I still get this error:

 

2013-04-30 07:43:02,520 FATAL org.apache.hadoop.mapred.JobTracker:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

 

 

From: Mohammad Tariq [mailto:donta...@gmail.com] 
Sent: Monday, April 29, 2013 5:10 PM
To: user@hadoop.apache.org
Subject: Re: Incompartible cluserIDS

 

make it 755.




Warm Regards,

Tariq

https://mtariq.jux.com/

cloudfront.blogspot.com

 

 

 

RE: Permission problem

[Kevin Burton]

 

Thank you.

 

mapred.system.dir is not set. I am guessing that it is whatever the default
is. What should I set it to?

 

/tmp is already 777

 

kevin@devUbuntu05:~$ hadoop fs -ls /tmp

Found 1 items

drwxr-xr-x   - hdfs supergroup  0 2013-04-29 15:45 /tmp/mapred

kevin@devUbuntu05:~$ hadoop fs -ls -d /tmp

Found 1 items

drwxrwxrwt   - hdfs supergroup  0 2013-04-29 15:45 /tmp

 

But notice that the mapred folder in the /tmp folder is 755.

So I changed it:

 

kevin@devUbuntu05 $ hadoop fs -ls -d /tmp

drwxrwxrwt   - hdfs supergroup  0 2013-04-29 15:45 /tmp

 

kevin@devUbuntu05 $ hadoop fs -ls -R /tmp 

drwxr-xr-x   - mapred supergroup  0 2013-04-29 15:45 /tmp/mapred

drwxr-xr-x   - mapred supergroup  0 2013-04-29 15:45
/tmp/mapred/system

 

I still get the errors in the log file:

 

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Failed to
operate on mapred.system.dir
(hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system)
because of permissions.

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: This
directory should be owned by the user 'mapred (auth:SIMPLE)'

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Bailing
out ...

. . . . .

org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

. . . . .

Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessContr
olException): Permission denied: user=mapred, access=WRITE,
inode="/":hdfs:supergroup:drwxrwxr-x

2013-04-30 09:35:11,610 FATAL org.apache.hadoop.mapred.JobTracker:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

. . . . . .

 

 

From: Arpit Gupta [mailto:ar...@hortonworks.com] 
Sent: Tuesday, April 30, 2013 9:25 AM
To: user@hadoop.apache.org
Subject: Re: Permission problem

 

what is your mapred.system.dir set to in mapred-site.xml?

 

By default it will write to /tmp on hdfs.

 

So you can do the following

 

create /tmp on hdfs and chmod it to 777 as user hdfs and then restart
jobtracker and tasktrackers.

 

In case its set to /mapred/something then create /mapred and chown it to
user mapred.

 


--
Arpit Gupta

Hortonworks Inc.
http://hortonworks.com/

 

On Apr 30, 2013, at 6:36 AM, "Kevin Burton" 
wrote:





To further complicate the issue the log file in
(/var/log/hadoop-0.20-mapreduce/hadoop-hadoop-jobtracker-devUbuntu05.log) is
owned by mapred:mapred and the name of the file seems to indicate some other
lineage (hadoop,hadoop). I am out of my league in understanding the
permission structure for hadoop hdfs and mr. Ideas?

 

From: Kevin Burton [mailto:rkevinburton@ <http://charter.net> charter.net] 
Sent: Tuesday, April 30, 2013 8:31 AM
To:  <mailto:user@hadoop.apache.org> user@hadoop.apache.org
Cc: 'Mohammad Tariq'
Subject: RE: Permission problem

 

That is what I perceive as the problem. The hdfs file system was created
with the user 'hdfs' owning the root ('/') but for some reason with a M/R
job the user 'mapred' needs to have write permission to the root. I don't
know how to satisfy both conditions. That is one reason that I relaxed the
permission to 775 so that the group would also have write permission but
that didn't seem to help.

 

From: Mohammad Tariq [ <mailto:donta...@gmail.com>
mailto:donta...@gmail.com] 
Sent: Tuesday, April 30, 2013 8:20 AM
To: Kevin Burton
Subject: Re: Permission problem

 

user?"ls" shows "hdfs" and the log says "mapred"..




Warm Regards,

Tariq

 <https://mtariq.jux.com/> https://mtariq.jux.com/

 <http://cloudfront.blogspot.com> cloudfront.blogspot.com

 

On Tue, Apr 30, 2013 at 6:22 PM, Kevin Burton <
<mailto:rkevinbur...@charter.net> rkevinbur...@charter.net> wrote:

I have relaxed it even further so now it is 775

 

 <mailto:kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$>
kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$ hadoop fs -ls -d /

Found 1 items

drwxrwxr-x   - hdfs supergroup  0 2013-04-29 15:43 /

 

But I still get this error:

 

2013-04-30 07:43:02,520 FATAL org.apache.hadoop.mapred.JobTracker:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

 

 

From: Mohammad Tariq [mailto: <mailto:donta...@gmail.com>
donta...@gmail.com] 
Sent: Monday, April 29, 2013 5:10 PM
To:  <mailto:user@hadoop.apache.org> user@hadoop.apache.org
Subject: Re: Incompartible cluserIDS

 

make it 755.




Warm Regards,

Tariq

 <https://mtariq.jux.com/> https://mtariq.jux.com/

 <http://cloudfront.blogspot.com> cloudfront.blogspot.com

 

 

 

RE: Permission problem

[Kevin Burton]

In core-site.xml I have:

 



  fs.default.name

  hdfs://devubuntu05:9000

  The name of the default file system. A URI whose scheme and
authority determine the FileSystem implementation. 



 

In hdfs-site.xml I have

 



  hadoop.tmp.dir

  /data/hadoop/tmp/hadoop-${user.name}

  Hadoop temporary folder



 

 

From: Arpit Gupta [mailto:ar...@hortonworks.com] 
Sent: Tuesday, April 30, 2013 9:48 AM
To: Kevin Burton
Cc: user@hadoop.apache.org
Subject: Re: Permission problem

 

Based on the logs your system dir is set to

 

 
hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system

 

 

what is your fs.default.name and hadoop.tmp.dir in core-site.xml set to?

 

 


--
Arpit Gupta

Hortonworks Inc.
http://hortonworks.com/

 

On Apr 30, 2013, at 7:39 AM, "Kevin Burton" 
wrote:





 

Thank you.

 

mapred.system.dir is not set. I am guessing that it is whatever the default
is. What should I set it to?

 

/tmp is already 777

 

kevin@devUbuntu05:~$ hadoop fs -ls /tmp

Found 1 items

drwxr-xr-x   - hdfs supergroup  0 2013-04-29 15:45 /tmp/mapred

kevin@devUbuntu05:~$ hadoop fs -ls -d /tmp

Found 1 items

drwxrwxrwt   - hdfs supergroup  0 2013-04-29 15:45 /tmp

 

But notice that the mapred folder in the /tmp folder is 755.

So I changed it:

 

kevin@devUbuntu05 $ hadoop fs -ls -d /tmp

drwxrwxrwt   - hdfs supergroup  0 2013-04-29 15:45 /tmp

 

kevin@devUbuntu05 $ hadoop fs -ls -R /tmp

drwxr-xr-x   - mapred supergroup  0 2013-04-29 15:45 /tmp/mapred

drwxr-xr-x   - mapred supergroup  0 2013-04-29 15:45
/tmp/mapred/system

 

I still get the errors in the log file:

 

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Failed to
operate on mapred.system.dir (

hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system) because
of permissions.

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: This
directory should be owned by the user 'mapred (auth:SIMPLE)'

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Bailing
out ...

. . . . .

org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

. . . . .

Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessContr
olException): Permission denied: user=mapred, access=WRITE,
inode="/":hdfs:supergroup:drwxrwxr-x

2013-04-30 09:35:11,610 FATAL org.apache.hadoop.mapred.JobTracker:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

. . . . . .

 

 

From: Arpit Gupta [mailto:ar...@hortonworks.com] 
Sent: Tuesday, April 30, 2013 9:25 AM
To: user@hadoop.apache.org
Subject: Re: Permission problem

 

what is your mapred.system.dir set to in mapred-site.xml?

 

By default it will write to /tmp on hdfs.

 

So you can do the following

 

create /tmp on hdfs and chmod it to 777 as user hdfs and then restart
jobtracker and tasktrackers.

 

In case its set to /mapred/something then create /mapred and chown it to
user mapred.

 


--
Arpit Gupta

Hortonworks Inc.
 <http://hortonworks.com/> http://hortonworks.com/

 

On Apr 30, 2013, at 6:36 AM, "Kevin Burton" <
<mailto:rkevinbur...@charter.net> rkevinbur...@charter.net> wrote:






To further complicate the issue the log file in
(/var/log/hadoop-0.20-mapreduce/hadoop-hadoop-jobtracker-devUbuntu05.log) is
owned by mapred:mapred and the name of the file seems to indicate some other
lineage (hadoop,hadoop). I am out of my league in understanding the
permission structure for hadoop hdfs and mr. Ideas?

 

From: Kevin Burton [mailto:rkevinburton@ <http://charter.net> charter.net] 
Sent: Tuesday, April 30, 2013 8:31 AM
To:  <mailto:user@hadoop.apache.org> user@hadoop.apache.org
Cc: 'Mohammad Tariq'
Subject: RE: Permission problem

 

That is what I perceive as the problem. The hdfs file system was created
with the user 'hdfs' owning the root ('/') but for some reason with a M/R
job the user 'mapred' needs to have write permission to the root. I don't
know how to satisfy both conditions. That is one reason that I relaxed the
permission to 775 so that the group would also have write permission but
that didn't seem to help.

 

From: Mohammad Tariq [ <mailto:donta...@gmail.com>
mailto:donta...@gmail.com] 
Sent: Tuesday, April 30, 2013 8:20 AM
To: Kevin Burton
Subject: Re: Permission problem

 

user?"ls" shows "hdfs" and the log says "mapred"..




Warm Regards,

Tariq

 <https://mtariq.jux.com/> https://mtariq.jux.com/

 <http://cloudfront.blogspot.com> cloudfront.blogspot.com

 

On Tue, Apr 30, 2013 at 6:22 PM, Kevin Burton <
<mailto:rkevinbur...@charter.net> rkevinbur...@charter.net> wrote:

I have relaxed i

RE: Permission problem

[Kevin Burton]

I am not clear on what you are suggesting to create on HDFS or the local
file system. As I understand it hadoop.tmp.dir is the local file system. I
changed it so that the temporary files would be on a disk that has more
capacity then /tmp. So you are suggesting that I create /data/hadoop/tmp on
HDFS. I already have this created.

 

Found 1 items

drwxr-xr-x   - mapred supergroup  0 2013-04-29 15:45 /tmp/mapred

kevin@devUbuntu05:/etc/hadoop/conf$ hadoop fs -ls -d /tmp

Found 1 items

drwxrwxrwt   - hdfs supergroup  0 2013-04-29 15:45 /tmp

 

When you suggest that I 'chmod -R 777 /data'. You are suggesting that I open
up all the data to everyone? Isn't that a bit extreme? First /data is the
mount point for this drive and there are other uses for this drive than
hadoop so there are other folders. That is why there is /data/hadoop. As far
as hadoop is concerned:

 

kevin@devUbuntu05:/etc/hadoop/conf$ ls -l /data/hadoop/

total 12

drwxrwxr-x 4 hdfs   hadoop 4096 Apr 29 16:38 dfs

drwxrwxr-x 3 mapred hadoop 4096 Apr 29 11:33 mapred

drwxrwxrwx 3 hdfs   hadoop 4096 Apr 19 15:14 tmp

 

dfs would be where the data blocks for the hdfs file system would go, mapred
would be the folder for M/R jobs, and tmp would be temporary storage. These
are all on the local file system. Do I have to make all of this read-write
for everyone in order to get it to work?

 

From: Arpit Gupta [mailto:ar...@hortonworks.com] 
Sent: Tuesday, April 30, 2013 10:01 AM
To: user@hadoop.apache.org
Subject: Re: Permission problem

 

ah 

 

this is what mapred.sytem.dir defaults to

 



  mapred.system.dir

  ${hadoop.tmp.dir}/mapred/system

  The directory where MapReduce stores control files.

  



 

 

So thats why its trying to write to
/data/hadoop/tmp/hadoop-mapred/mapred/system
 

 

 

So if you want hadoop.tmp.dir to be /data/hadoop/tmp/hadoop-${user.name}
then i would suggest that create /data/hadoop/tmp on hdfs and chmod -R 777
/data or you can remove the hadoop.tmp.dir from your configs and let it be
set to the default value of

 



  hadoop.tmp.dir

  /tmp/hadoop-${user.name}

  A base for other temporary directories.



 

So to fix your problem you can do the above or set mapred.system.dir to
/tmp/mapred/system in your mapred-site.xml.


--
Arpit Gupta

Hortonworks Inc.
http://hortonworks.com/

 

On Apr 30, 2013, at 7:55 AM, "Kevin Burton" 
wrote:





In core-site.xml I have:

 



  fs.default.name

   
hdfs://devubuntu05:9000

  The name of the default file system. A URI whose scheme and
authority determine the FileSystem implementation. 



 

In hdfs-site.xml I have

 



  hadoop.tmp.dir

  /data/hadoop/tmp/hadoop-${user.name}

  Hadoop temporary folder



 

 

From: Arpit Gupta [mailto:ar...@hortonworks.com] 
Sent: Tuesday, April 30, 2013 9:48 AM
To: Kevin Burton
Cc: user@hadoop.apache.org
Subject: Re: Permission problem

 

Based on the logs your system dir is set to

 

 
hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system

 

 

what is your fs.default.name and hadoop.tmp.dir in core-site.xml set to?

 

 


--
Arpit Gupta

Hortonworks Inc.
 <http://hortonworks.com/> http://hortonworks.com/

 

On Apr 30, 2013, at 7:39 AM, "Kevin Burton" <
<mailto:rkevinbur...@charter.net> rkevinbur...@charter.net> wrote:






 

Thank you.

 

mapred.system.dir is not set. I am guessing that it is whatever the default
is. What should I set it to?

 

/tmp is already 777

 

kevin@devUbuntu05:~$ hadoop fs -ls /tmp

Found 1 items

drwxr-xr-x   - hdfs supergroup  0 2013-04-29 15:45 /tmp/mapred

kevin@devUbuntu05:~$ hadoop fs -ls -d /tmp

Found 1 items

drwxrwxrwt   - hdfs supergroup  0 2013-04-29 15:45 /tmp

 

But notice that the mapred folder in the /tmp folder is 755.

So I changed it:

 

kevin@devUbuntu05 $ hadoop fs -ls -d /tmp

drwxrwxrwt   - hdfs supergroup  0 2013-04-29 15:45 /tmp

 

kevin@devUbuntu05 $ hadoop fs -ls -R /tmp

drwxr-xr-x   - mapred supergroup  0 2013-04-29 15:45 /tmp/mapred

drwxr-xr-x   - mapred supergroup  0 2013-04-29 15:45
/tmp/mapred/system

 

I still get the errors in the log file:

 

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Failed to
operate on mapred.system.dir (

hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system) because
of permissions.

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: This
directory should be owned by the user 'mapred (auth:SIMPLE)'

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Bailing
out ...

. . . . .

org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

. . . . .

Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessContr
olException): Permission denied: user=mapred, access=WRITE,
inode="/":hdfs:super

RE: Permission problem

[Kevin Burton]

I am not sure how to create a jira.

 

Again I am not sure I understand your workaround. You are suggesting that I
create /data/hadoop/tmp on HDFS like:

 

sudo -u hdfs hadoop fs -mkdir /data/hadoop/tmp

 

I don't think I can chmod -R 777 on /data since it is a disk and as I
indicated it is being used to store data other than that used by hadoop.
Even chmod -R 777 on /data/hadoop seems extreme as there is a dfs, mapred,
and tmp folder. Which one of these local folders need to be opened up? I
would rather not open up all folders to the world if at all possible.

 

From: Arpit Gupta [mailto:ar...@hortonworks.com] 
Sent: Tuesday, April 30, 2013 10:48 AM
To: Kevin Burton
Cc: user@hadoop.apache.org
Subject: Re: Permission problem

 

It looks like hadoop.tmp.dir is being used both for local and hdfs
directories. Can you create a jira for this?

 

What i recommended is that you create /data/hadoop/tmp on hdfs and chmod -R
/data 

 

--
Arpit Gupta

Hortonworks Inc.
http://hortonworks.com/

 

On Apr 30, 2013, at 8:22 AM, "Kevin Burton" 
wrote:





I am not clear on what you are suggesting to create on HDFS or the local
file system. As I understand it hadoop.tmp.dir is the local file system. I
changed it so that the temporary files would be on a disk that has more
capacity then /tmp. So you are suggesting that I create /data/hadoop/tmp on
HDFS. I already have this created.

 

Found 1 items

drwxr-xr-x   - mapred supergroup  0 2013-04-29 15:45 /tmp/mapred

kevin@devUbuntu05:/etc/hadoop/conf$ hadoop fs -ls -d /tmp

Found 1 items

drwxrwxrwt   - hdfs supergroup  0 2013-04-29 15:45 /tmp

 

When you suggest that I 'chmod -R 777 /data'. You are suggesting that I open
up all the data to everyone? Isn't that a bit extreme? First /data is the
mount point for this drive and there are other uses for this drive than
hadoop so there are other folders. That is why there is /data/hadoop. As far
as hadoop is concerned:

 

kevin@devUbuntu05:/etc/hadoop/conf$ ls -l /data/hadoop/

total 12

drwxrwxr-x 4 hdfs   hadoop 4096 Apr 29 16:38 dfs

drwxrwxr-x 3 mapred hadoop 4096 Apr 29 11:33 mapred

drwxrwxrwx 3 hdfs   hadoop 4096 Apr 19 15:14 tmp

 

dfs would be where the data blocks for the hdfs file system would go, mapred
would be the folder for M/R jobs, and tmp would be temporary storage. These
are all on the local file system. Do I have to make all of this read-write
for everyone in order to get it to work?

 

From: Arpit Gupta [mailto:ar...@hortonworks.com] 
Sent: Tuesday, April 30, 2013 10:01 AM
To: user@hadoop.apache.org
Subject: Re: Permission problem

 

ah 

 

this is what mapred.sytem.dir defaults to

 



  mapred.system.dir

  ${hadoop.tmp.dir}/mapred/system

  The directory where MapReduce stores control files.

  



 

 

So thats why its trying to write to

/data/hadoop/tmp/hadoop-mapred/mapred/system

 

 

So if you want hadoop.tmp.dir to be /data/hadoop/tmp/hadoop-${user.name}
then i would suggest that create /data/hadoop/tmp on hdfs and chmod -R 777
/data or you can remove the hadoop.tmp.dir from your configs and let it be
set to the default value of

 



  hadoop.tmp.dir

  /tmp/hadoop-${user.name}

  A base for other temporary directories.



 

So to fix your problem you can do the above or set mapred.system.dir to
/tmp/mapred/system in your mapred-site.xml.


--
Arpit Gupta

Hortonworks Inc.
 <http://hortonworks.com/> http://hortonworks.com/

 

On Apr 30, 2013, at 7:55 AM, "Kevin Burton" <
<mailto:rkevinbur...@charter.net> rkevinbur...@charter.net> wrote:






In core-site.xml I have:

 



  fs.default.name

   
hdfs://devubuntu05:9000

  The name of the default file system. A URI whose scheme and
authority determine the FileSystem implementation. 



 

In hdfs-site.xml I have

 



  hadoop.tmp.dir

  /data/hadoop/tmp/hadoop-${user.name}

  Hadoop temporary folder



 

 

From: Arpit Gupta [mailto:arpit@ <http://hortonworks.com> hortonworks.com] 
Sent: Tuesday, April 30, 2013 9:48 AM
To: Kevin Burton
Cc:  <mailto:user@hadoop.apache.org> user@hadoop.apache.org
Subject: Re: Permission problem

 

Based on the logs your system dir is set to

 

 
hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system

 

 

what is your fs.default.name and hadoop.tmp.dir in core-site.xml set to?

 

 


--
Arpit Gupta

Hortonworks Inc.
 <http://hortonworks.com/> http://hortonworks.com/

 

On Apr 30, 2013, at 7:39 AM, "Kevin Burton" <
<mailto:rkevinbur...@charter.net> rkevinbur...@charter.net> wrote:







 

Thank you.

 

mapred.system.dir is not set. I am guessing that it is whatever the default
is. What should I set it to?

 

/tmp is already 777

 

kevin@devUbuntu05:~$ hadoop fs -ls /tmp

Found 1 items

drwxr-xr-x   - hdfs supergroup  0 2013-04-29 15:45 /tmp/mapred

kevin@devUbuntu05:~$ hadoop fs -ls -d /tmp

Found 1 items

drwxrwxrwt   - hdfs supergroup  0 2013-

Can't initialize cluster

[Kevin Burton]

I have a simple MapReduce job that I am trying to get to run on my cluster.
When I run it I get:

 

13/04/30 11:27:45 INFO mapreduce.Cluster: Failed to use
org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid
"mapreduce.jobtracker.address" configuration value for LocalJobRunner :
"devubuntu05:9001"

13/04/30 11:27:45 ERROR security.UserGroupInformation:
PriviledgedActionException as:kevin (auth:SIMPLE) cause:java.io.IOException:
Cannot initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.

Exception in thread "main" java.io.IOException: Cannot initialize Cluster.
Please check your configuration for mapreduce.framework.name and the
correspond server addresses.

 

My core-site.xml looks like:

 



  fs.default.name

  hdfs://devubuntu05:9000

  The name of the default file system. A URI whose scheme and
authority determine the FileSystem implementation. 



 

So I am unclear as to why it is looking at devubuntu05:9001?

 

Here is the code:

 

public static void WordCount( String[] args )  throws Exception {

Configuration conf = new Configuration();

String[] otherArgs = new GenericOptionsParser(conf,
args).getRemainingArgs();

if (otherArgs.length != 2) {

System.err.println("Usage: wordcount  ");

System.exit(2);

}

Job job = new Job(conf, "word count");

job.setJarByClass(WordCount.class);

job.setMapperClass(WordCount.TokenizerMapper.class);

job.setCombinerClass(WordCount.IntSumReducer.class);

job.setReducerClass(WordCount.IntSumReducer.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(IntWritable.class);

 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(job, new
Path(otherArgs[0]));

 
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job,
new Path(otherArgs[1]));

System.exit(job.waitForCompletion(true) ? 0 : 1);

 

Ideas?

RE: Can't initialize cluster

[Kevin Burton]

To be clear when this code is run with 'java -jar' it runs without
exception. The exception occurs when I run with 'hadoop jar'.

 

From: Kevin Burton [mailto:rkevinbur...@charter.net] 
Sent: Tuesday, April 30, 2013 11:36 AM
To: user@hadoop.apache.org
Subject: Can't initialize cluster

 

I have a simple MapReduce job that I am trying to get to run on my cluster.
When I run it I get:

 

13/04/30 11:27:45 INFO mapreduce.Cluster: Failed to use
org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid
"mapreduce.jobtracker.address" configuration value for LocalJobRunner :
"devubuntu05:9001"

13/04/30 11:27:45 ERROR security.UserGroupInformation:
PriviledgedActionException as:kevin (auth:SIMPLE) cause:java.io.IOException:
Cannot initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.

Exception in thread "main" java.io.IOException: Cannot initialize Cluster.
Please check your configuration for mapreduce.framework.name and the
correspond server addresses.

 

My core-site.xml looks like:

 



  fs.default.name

  hdfs://devubuntu05:9000

  The name of the default file system. A URI whose scheme and
authority determine the FileSystem implementation. 



 

So I am unclear as to why it is looking at devubuntu05:9001?

 

Here is the code:

 

public static void WordCount( String[] args )  throws Exception {

Configuration conf = new Configuration();

String[] otherArgs = new GenericOptionsParser(conf,
args).getRemainingArgs();

if (otherArgs.length != 2) {

System.err.println("Usage: wordcount  ");

System.exit(2);

}

Job job = new Job(conf, "word count");

job.setJarByClass(WordCount.class);

job.setMapperClass(WordCount.TokenizerMapper.class);

job.setCombinerClass(WordCount.IntSumReducer.class);

job.setReducerClass(WordCount.IntSumReducer.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(IntWritable.class);

 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(job, new
Path(otherArgs[0]));

 
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job,
new Path(otherArgs[1]));

System.exit(job.waitForCompletion(true) ? 0 : 1);

 

Ideas?

RE: Can't initialize cluster

[Kevin Burton]

We/I are/am making progress. Now I get the error:

 

13/04/30 12:59:40 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.

13/04/30 12:59:40 INFO mapred.JobClient: Cleaning up the staging area
hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/staging/kevin/.
staging/job_201304301251_0003

13/04/30 12:59:40 ERROR security.UserGroupInformation:
PriviledgedActionException as:kevin (auth:SIMPLE)
cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input
path does not exist: hdfs://devubuntu05:9000/user/kevin/input

Exception in thread "main"
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does
not exist: hdfs://devubuntu05:9000/user/kevin/input

 

When I run it with java -jar the input and output is the local folder. When
running it with hadoop jar it seems to be expecting the folders (input and
output) to be on the HDFS file system. I am not sure why these two methods
of invocation don't make the same file system assumptions.

 

It is

 

hadoop jar WordCount.jar input output (which gives the above exception)

 

versus

 

java -jar WordCount.jar input output (which outputs the wordcount statistics
to the output folder)

 

This is run in the local /home/kevin/WordCount folder.

 

Kevin

 

From: Mohammad Tariq [mailto:donta...@gmail.com] 
Sent: Tuesday, April 30, 2013 12:33 PM
To: user@hadoop.apache.org
Subject: Re: Can't initialize cluster

 

Set "HADOOP_MAPRED_HOME" in your hadoop-env.sh file and re-run the job. See
if it helps.




Warm Regards,

Tariq

https://mtariq.jux.com/

cloudfront.blogspot.com

 

On Tue, Apr 30, 2013 at 10:10 PM, Kevin Burton 
wrote:

To be clear when this code is run with 'java -jar' it runs without
exception. The exception occurs when I run with 'hadoop jar'.

 

From: Kevin Burton [mailto:rkevinbur...@charter.net] 
Sent: Tuesday, April 30, 2013 11:36 AM
To: user@hadoop.apache.org
Subject: Can't initialize cluster

 

I have a simple MapReduce job that I am trying to get to run on my cluster.
When I run it I get:

 

13/04/30 11:27:45 INFO mapreduce.Cluster: Failed to use
org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid
"mapreduce.jobtracker.address" configuration value for LocalJobRunner :
"devubuntu05:9001"

13/04/30 11:27:45 ERROR security.UserGroupInformation:
PriviledgedActionException as:kevin (auth:SIMPLE) cause:java.io.IOException:
Cannot initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.

Exception in thread "main" java.io.IOException: Cannot initialize Cluster.
Please check your configuration for mapreduce.framework.name and the
correspond server addresses.

 

My core-site.xml looks like:

 



  fs.default.name

  hdfs://devubuntu05:9000

  The name of the default file system. A URI whose scheme and
authority determine the FileSystem implementation. 



 

So I am unclear as to why it is looking at devubuntu05:9001?

 

Here is the code:

 

public static void WordCount( String[] args )  throws Exception {

Configuration conf = new Configuration();

String[] otherArgs = new GenericOptionsParser(conf,
args).getRemainingArgs();

if (otherArgs.length != 2) {

System.err.println("Usage: wordcount  ");

System.exit(2);

}

Job job = new Job(conf, "word count");

job.setJarByClass(WordCount.class);

job.setMapperClass(WordCount.TokenizerMapper.class);

job.setCombinerClass(WordCount.IntSumReducer.class);

job.setReducerClass(WordCount.IntSumReducer.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(IntWritable.class);

 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(job, new
Path(otherArgs[0]));

 
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job,
new Path(otherArgs[1]));

System.exit(job.waitForCompletion(true) ? 0 : 1);

 

Ideas?

 

RE: Can't initialize cluster

[Kevin Burton]

Tariq,

 

Thank you. I tried this and the summary of the map reduce job looks like:

 

13/04/30 14:02:35 INFO mapred.JobClient: Job complete: job_201304301251_0004

13/04/30 14:02:35 INFO mapred.JobClient: Counters: 7

13/04/30 14:02:35 INFO mapred.JobClient:   Job Counters

13/04/30 14:02:35 INFO mapred.JobClient: Failed map tasks=1

13/04/30 14:02:35 INFO mapred.JobClient: Launched map tasks=27

13/04/30 14:02:35 INFO mapred.JobClient: Rack-local map tasks=27

13/04/30 14:02:35 INFO mapred.JobClient: Total time spent by all maps in
occupied slots (ms)=151904

13/04/30 14:02:35 INFO mapred.JobClient: Total time spent by all reduces
in occupied slots (ms)=0

13/04/30 14:02:35 INFO mapred.JobClient: Total time spent by all maps
waiting after reserving slots (ms)=0

13/04/30 14:02:35 INFO mapred.JobClient: Total time spent by all reduces
waiting after reserving slots (ms)=0

 

But there were a number of exceptions thrown and it seemed to take longer
than just running it standalone (I should have at least 4 machines working
on this). The exceptions are my main concern now:

 

(there were quite a few)

. . . . .

13/04/30 14:02:27 INFO mapred.JobClient: Task Id :
attempt_201304301251_0004_m_05_1, Status : FAILED

java.io.FileNotFoundException: File
file:/home/kevin/WordCount/input/hadoop-core-2.0.0-mr1-cdh4.2.1.jar does not
exist

. . . . 

13/04/30 14:02:28 INFO mapred.JobClient: Task Id :
attempt_201304301251_0004_m_06_1, Status : FAILED

java.io.FileNotFoundException: File
file:/home/kevin/WordCount/input/guava-11.0.2.jar does not exist

. . . .

13/04/30 14:02:28 INFO mapred.JobClient: Task Id :
attempt_201304301251_0004_m_08_0, Status : FAILED

java.io.FileNotFoundException: File
file:/home/kevin/WordCount/input/zookeeper-3.4.5-cdh4.2.1.jar does not exist

. . . . .

13/04/30 14:02:28 INFO mapred.JobClient: Task Id :
attempt_201304301251_0004_m_01_2, Status : FAILED

java.io.FileNotFoundException: File
file:/home/kevin/WordCount/input/tools.jar does not exist

. . . . .

13/04/30 14:02:28 INFO mapred.JobClient: Task Id :
attempt_201304301251_0004_m_00_2, Status : FAILED

java.io.FileNotFoundException: File
file:/home/kevin/WordCount/input/Websters.txt does not exist 

. . . .

13/04/30 14:02:33 INFO mapred.JobClient: Task Id :
attempt_201304301251_0004_m_02_2, Status : FAILED

java.io.FileNotFoundException: File
file:/home/kevin/WordCount/input/hadoop-hdfs-2.0.0-cdh4.2.1.jar does not
exist

. . . . 

13/04/30 14:02:33 INFO mapred.JobClient: Task Id :
attempt_201304301251_0004_m_04_2, Status : FAILED

java.io.FileNotFoundException: File
file:/home/kevin/WordCount/input/hadoop-common-2.0.0-cdh4.2.1.jar does not
exist

.  . . .

13/04/30 14:02:33 INFO mapred.JobClient: Task Id :
attempt_201304301251_0004_m_03_2, Status : FAILED

java.io.FileNotFoundException: File
file:/home/kevin/WordCount/input/core-3.1.1.jar does not exist

 

No output folder was created (probably because of the numerous errors).

 

Kevin

 

From: Mohammad Tariq [mailto:donta...@gmail.com] 
Sent: Tuesday, April 30, 2013 1:32 PM
To: Kevin Burton
Subject: Re: Can't initialize cluster

 

Hello again Kevin,

 

 Good that you are making progress. This is happening because when you
are running it as a hadoop job, it looks for the the files in HDFS and when
you run it as a job program it looks into the local FS. Use this as your
input in your code and see if it helps : 

 

file:///home/kevin/input  

 




Warm Regards,

Tariq

https://mtariq.jux.com/

cloudfront.blogspot.com

 

On Tue, Apr 30, 2013 at 11:36 PM, Kevin Burton 
wrote:

We/I are/am making progress. Now I get the error:

 

13/04/30 12:59:40 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.

13/04/30 12:59:40 INFO mapred.JobClient: Cleaning up the staging area
hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/staging/kevin/.
staging/job_201304301251_0003

13/04/30 12:59:40 ERROR security.UserGroupInformation:
PriviledgedActionException as:kevin (auth:SIMPLE)
cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input
path does not exist: hdfs://devubuntu05:9000/user/kevin/input

Exception in thread "main"
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does
not exist: hdfs://devubuntu05:9000/user/kevin/input

 

When I run it with java -jar the input and output is the local folder. When
running it with hadoop jar it seems to be expecting the folders (input and
output) to be on the HDFS file system. I am not sure why these two methods
of invocation don't make the same file system assumptions.

 

It is

 

hadoop jar WordCount.jar input output (which gives the above exception)

 

versus

 

java -jar WordCount.jar input output (which outputs the wordcount statistics
to the output folder)

 

This is run in the local /home/kevin/WordCount folder.

 

Kevin

 

Fwd: class not found on namenode/datanode startup

[Kevin D'Elia]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1


Hello,

I have configured hadoop install according to instructions I found on
the internet; when I start hadoop namenode/datanode, I get:

java.lang.NoClassDefFoundError:
org/apache/hadoop/hdfs/server/namenode/NameNode
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hdfs.server.namenode.NameNode
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class:
org.apache.hadoop.hdfs.server.namenode.NameNode.  Program will exit.
Exception in thread "main" ulimit -a for user kdelia
core file size  (blocks, -c) unlimited
data seg size   (kbytes, -d) unlimited
file size   (blocks, -f) unlimited
open files  (-n) 256
pipe size(512 bytes, -p) 8
stack size  (kbytes, -s) 2024
cpu time   (seconds, -t) unlimited
max user processes  (-u) 256
virtual memory  (kbytes, -v) unlimited


hadoop classpath shows me:

/home/kdelia/yarn/hadoop-2.2.0/etc/hadoop:/home/kdelia/yarn/hadoop-2.2.0/share/hadoop/common/lib/*:/home/kdelia/yarn/hadoop-2.2.0/share/hadoop/common/*:/home/kdelia/yarn/hadoop-2.2.0/share/hadoop/hdfs:/home/kdelia/yarn/hadoop-2.2.0/share/hadoop/hdfs/lib/*:/home/kdelia/yarn/hadoop-2.2.0/share/hadoop/hdfs/*:/home/kdelia/yarn/hadoop-2.2.0/share/hadoop/yarn/lib/*:/home/kdelia/yarn/hadoop-2.2.0/share/hadoop/yarn/*:/home/kdelia/yarn/hadoop-2.2.0/share/hadoop/mapreduce/lib/*:/home/kdelia/yarn/hadoop-2.2.0/share/hadoop/mapreduce/*:/home/kdelia/yarn/hadoop-2.2.0/contrib/capacity-scheduler/*.jar


So, it appears that the libraries needed by the services can be found.
 Not sure why I am getting the class not found exception.

I am using instructions from this site:

http://raseshmori.wordpress.com/2012/09/23/install-hadoop-2-0-1-yarn-nextgen/

Thanks!
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSi27WAAoJEIqFaT7R0oyMviwH/j+7398y+e+iHUILpPdxa4UW
gYYh5OBPk+Ae8N1I+a5OsUOoZxxkcET2vZIChgbRe/S3k8xullrUYj5Y4rLHT8AF
mDL/7U2+hpCX3E1TJeCetI/uyVgqtVsEyU6pwocYmnc7zIh3YpzFvahO40q7Iik9
4I5AHi6yFfmvHxdni/ZOgf/B8NS8NLVhmHxZcRFliXYZVyJsBTBpacTq8BY49i8n
gGod7eIu/NZ6HuYJqn+vfrPhopfMFVKOAWy510KlbAarFm22KXOQTHciiIAqML6O
+ChhE5mz1G2Oo5qYMvoi0/TxT+vTC5rOGCZ6sQn3/R4s0NvXJOwbdynSJGWSnbc=
=2fuf
-END PGP SIGNATURE-

Re: Suggestions required for learning Hadoop

[Kevin O'dell]

This might seem a little bias:

http://www.cloudera.com/resources/

http://www.cloudera.com/resources/training/

http://university.cloudera.com/



On Thu, Sep 13, 2012 at 2:27 PM,  wrote:

>
> There's the MapR academy:
>
> http://academy.mapr.com/
>
>
>  Original Message 
> Subject: Suggestions required for learning Hadoop
> From: "Munnavar Shaik" 
> Date: Thu, September 13, 2012 1:37 pm
> To: 
>
> Dear Team Members,
>
> I am working as a Linux Administrator, I am interested to work on
> Hadoop. Please let me know from where and how I can start to learning.
> It is very great full to help for learning Hadoop and its related
> project.
>
> Thank you Team,
> Munnavar
>



-- 
Kevin O'Dell
Customer Operations Engineer, Cloudera

Re: HDFS using SAN

[Kevin O'dell]

 of magnitude more of iops we care about for
> now, but given it is a shared infrastructure and we may expand our data
> size, it may not be an advantage in the future.
>
>  
>
> Another thing I am interested in: for MR jobs, where data locality is the
> key driver, how does that span out when using a SAN instead of direct
> storage?
>
>  
>
> And of course on the subjective topics of availability and reliability on
> using a SAN for data storage in HDFS, I would love to receive your views.*
> ***
>
>  
>
> Thanks,
>
> Abhishek
>
>  
>
>
>
> 
>
> ** **
>
> --
> Have a Nice Day!
> Lohit
>



-- 
Kevin O'Dell
Customer Operations Engineer, Cloudera

2.4 / yarn pig jobs fail due to exit code 1 from container.

[Kevin Burton]

Trying to track down exactly what's happening.

Right now I'm getting this (see below).

The setup documentation for 2.4 could definitely be better.  Probably with
a sample/working config. Looks like too much of this is left up as an
exercise to the user.

2014-05-23 21:20:30,652 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
launchContainer: [nice, -n, 0, bash,
/data/2/yarn/local/usercache/root/appcache/application_1400821083545_0004/container_1400821083545_0004_01_01/default_container_executor.sh]
2014-05-23 21:20:30,771 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit
code from container container_1400821083545_0004_01_01 is : 1
2014-05-23 21:20:30,771 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Exception from container-launch with container ID:
container_1400821083545_0004_01_01 and exit code: 1
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)


-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
Skype: *burtonator*
blog: http://burtonator.wordpress.com
… or check out my Google+
profile

War is peace. Freedom is slavery. Ignorance is strength. Corporations are
people.

debugging class path issues with containers.

[Kevin Burton]

What's the best way to debug yarn container issues?

I was going to try to tweak the script but it gets deleted after the job
fails.

Looks like I'm having an issue with the classpath.. I'm getting a basic
hadoop NCDFE on startup so I think it just has a broken class path.

but of course I need to fix that…

.. I'd like to just print the class path or maybe run a command before the
container is launched to check the permissions, etc.

Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/service/CompositeService
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:792)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.service.CompositeService
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more

-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
Skype: *burtonator*
blog: http://burtonator.wordpress.com
… or check out my Google+
profile

War is peace. Freedom is slavery. Ignorance is strength. Corporations are
people.

The documentation for permissions of ./bin/container-executor should be more clear.

[Kevin Burton]

This just bit me… spent half a day figuring it out!   :-(

The only way I was able to debug it was with

./bin/container-executor --checksetup

Once that stopped complaining my jobs were working ok.

this shouldn't have taken that much time… initial setup documentation could
be seriously improved.

another recommendation, provide template files.

-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
Skype: *burtonator*
blog: http://burtonator.wordpress.com
… or check out my Google+
profile

War is peace. Freedom is slavery. Ignorance is strength. Corporations are
people.

MapReduce data decompression using a custom codec

[POUPON Kevin]

Hello,

I developed a custom compression codec for Hadoop. Of course Hadoop is set to 
use my codec when compressing data.
For testing purposes, I use the following two commands:

Compression test command:
---
hadoop jar 
/opt/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/lib/hadoop//../hadoop-mapreduce/hadoop-streaming.jar
 -Dmapreduce.output.fileoutputformat.compress=true -input /originalFiles/ 
-output /compressedFiles/ -mapper cat -reducer cat


Decompression test command:
---
hadoop jar 
/opt/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/lib/hadoop//../hadoop-mapreduce/hadoop-streaming.jar
 -Dmapreduce.output.fileoutputformat.compress=false -input /compressedFiles/ 
-output /decompressedFiles/ -mapper cat -reducer cat


As you can see, both of them are quite similar: only the compression option 
changes and the input/output directories.

The first command compresses the input data then 'cat' (the Linux command, you 
know) it to the output file.
The second one decompresses the input  data (which are supposed to be 
compressed) then 'cat' it to the output file. As I understand, Hadoop is 
supposed to auto-detect compressed input data and decompress it using the right 
codec.

Those test compression and decompression work well when Hadoop is set to use a 
default codec, like BZip2 or Snappy.

However, when using my custom compression codec, only the compression works: 
the decompression is sluggish and triggers errors (Java heap space):

packageJobJar: [] 
[/opt/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/lib/hadoop-mapreduce/hadoop-streaming-2.3.0-cdh5.1.2.jar]
 /tmp/streamjob6475393520304432687.jar tmpDir=null
14/09/09 15:33:21 INFO client.RMProxy: Connecting to ResourceManager at 
bluga2/10.1.96.222:8032
14/09/09 15:33:22 INFO client.RMProxy: Connecting to ResourceManager at 
bluga2/10.1.96.222:8032
14/09/09 15:33:23 INFO mapred.FileInputFormat: Total input paths to process : 1
14/09/09 15:33:23 INFO mapreduce.JobSubmitter: number of splits:1
14/09/09 15:33:23 INFO mapreduce.JobSubmitter: Submitting tokens for job: 
job_1410264242020_0016
14/09/09 15:33:24 INFO impl.YarnClientImpl: Submitted application 
application_1410264242020_0016
14/09/09 15:33:24 INFO mapreduce.Job: The url to track the job: 
http://bluga2:8088/proxy/application_1410264242020_0016/
14/09/09 15:33:24 INFO mapreduce.Job: Running job: job_1410264242020_0016
14/09/09 15:33:30 INFO mapreduce.Job: Job job_1410264242020_0016 running in 
uber mode : false
14/09/09 15:33:30 INFO mapreduce.Job:  map 0% reduce 0%
14/09/09 15:35:12 INFO mapreduce.Job:  map 100% reduce 0%
14/09/09 15:35:13 INFO mapreduce.Job: Task Id : 
attempt_1410264242020_0016_m_00_0, Status : FAILED
Error: Java heap space
14/09/09 15:35:14 INFO mapreduce.Job:  map 0% reduce 0%
14/09/09 15:35:41 INFO mapreduce.Job: Task Id : 
attempt_1410264242020_0016_m_00_1, Status : FAILED
Error: Java heap space
14/09/09 15:36:02 INFO mapreduce.Job: Task Id : 
attempt_1410264242020_0016_m_00_2, Status : FAILED
Error: Java heap space
14/09/09 15:36:49 INFO mapreduce.Job:  map 100% reduce 0%
14/09/09 15:36:50 INFO mapreduce.Job:  map 100% reduce 100%
14/09/09 15:36:56 INFO mapreduce.Job: Job job_1410264242020_0016 failed with 
state FAILED due to: Task failed task_1410264242020_0016_m_00
Job failed as tasks failed. failedMaps:1 failedReduces:0

14/09/09 15:36:58 INFO mapreduce.Job: Counters: 9
   Job Counters
 Failed map tasks=4
 Launched map tasks=4
 Other local map tasks=3
 Data-local map tasks=1
 Total time spent by all maps in occupied slots (ms)=190606
 Total time spent by all reduces in occupied slots (ms)=0
 Total time spent by all map tasks (ms)=190606
 Total vcore-seconds taken by all map tasks=190606
 Total megabyte-seconds taken by all map tasks=195180544
14/09/09 15:36:58 ERROR streaming.StreamJob: Job not Successful!
Streaming Command Failed!

I already tried to increase the map maximum heap size 
(mapreduce.map.java.opts.max.heap's YARN property) from 1 GiB to 2 GiB but the 
decompression still doesn't work. By the way, I'm compressing and decompressing 
a small ~2MB file and use the latest Cloudera version.

I built a quick Java test environment to try to reproduce the Hadoop codec call 
(instantiating the codec, creating a new compression stream from it ...). I 
noticed that the decompression is an infinite loop where only the first block 
of compressed data is decompressed, infinitely. This could explain the above 
Java heap space error.

What am I doing wrong/what did I forget ? How could my codec decompress data 
without troubles?

Thank you for helping !

Kévin Poupon

Hive _SUCCESS flag

[Kevin Lasenby]

Hello Hadoop community,

Does anyone know if there is a way to re-enable the _SUCCESS flag feature for 
Hive > 0.9.0?  It appears Hive overrides the 
"mapreduce.fileoutputcommitter.marksuccessfuljobs" setting and there's no way 
to change it.

Hive JIRA: https://issues.apache.org/jira/browse/HIVE-3700 <-- created 2012 
with no updates
Discussion about this behavior: 
http://stackoverflow.com/questions/13017433/override-hadoops-mapreduce-fileoutputcommitter-marksuccessfuljobs-in-oozie

Thanks,


Kevin Lasenby

v2.8.0: Setting PID dir EnvVars: libexec/thing-config.sh or etc/hadoop/thing-env.sh ?

[Kevin Buckley]

I've noted in

https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/ClusterSetup.html

that it says

8<8<8<8<8<
See etc/hadoop/hadoop-env.sh for other examples.

Other useful configuration parameters that you can customize include:

HADOOP_PID_DIR - The directory where the daemons’ process id files
 are stored.

HADOOP_LOG_DIR - The directory where the daemons’ log files are
 stored. Log files are automatically created if
 they don’t exist.

HADOOP_HEAPSIZE / YARN_HEAPSIZE - The maximum amount of heapsize
 to use, in MB e.g. if the varibale is set to 1000
 the heap will be set to 1000MB. This is used to
 configure the heap size for the daemon. By
 default, the value is 1000. If you want to
 configure the values separately for each deamon
 you can use.

In most cases, you should specify the HADOOP_PID_DIR and HADOOP_LOG_DIR
directories such that they can only be written to by the users that
are going to run the hadoop daemons. Otherwise there is the potential
for a symlink attack.
8<8<8<8<8<

and I have recently had a need to move my PID-files from their default,
/tmp, location, so as to avoid an over-aggressive /tmp cleanup.


So, if I am in

 /path/to/hadoop-2.8.0/etc/hadoop

I can see

$ grep PID *
hadoop-env.cmd:set HADOOP_PID_DIR=%HADOOP_PID_DIR%
hadoop-env.cmd:set HADOOP_SECURE_DN_PID_DIR=%HADOOP_PID_DIR%
hadoop-env.sh:export HADOOP_PID_DIR=${HADOOP_PID_DIR}
hadoop-env.sh:export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
mapred-env.sh:#export HADOOP_MAPRED_PID_DIR= # The pid files are
stored. /tmp by default.
$

so clearly PID_DIR values could be overriden there, although I note that the

 yarn-env.sh

file, unlike its "hadoop" and "mapred" bretheren, doesn't contain
any PID_DIR default lines, nor even commented suggestion lines for

 YARN_PID_DIR

which is used in, for exmaple,

 hadoop-2.8.0/sbin/yarn-daemon.sh



However, I've also noted that many of the Hadoop sbin scripts
will source a related file from

 hadoop-2.8.0/libexec/

so for example, a desired value for the env var

 YARN_PID_DIR

might thus be set in either (or both ?) of

 hadoop-2.8.0/etc/hadoop/yarn-env.sh
 hadoop-2.8.0/libexec/yarn-config.sh

and similarly,

  HADOOP_PID_DIR

might be set in either (or both ?) of

 hadoop-2.8.0/libexec/hadoop-config.sh
 hadoop-2.8.0/etc/hadoop/hadoop-env.sh


So,

is there a "to be preferred" file choice, between those two,
within which to set certain classes of EnvVars ?


Any info/pointers welcome,
Kevin

---
Kevin M. Buckley

eScience Consultant
School of Engineering and Computer Science
Victoria University of Wellington
New Zealand

-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

Kerberised JobHistory Server not starting: User jhs trying to create the /mr-history/done directory

[Kevin Buckley]

 keytabs.

So is there something special about the jhs user ?

Or perhaps something special about the other keytab values ?

Any clues/insight welcome,
Kevin

---
Kevin M. Buckley

eScience Consultant
School of Engineering and Computer Science
Victoria University of Wellington
New Zealand

-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

Re: Kerberised JobHistory Server not starting: User jhs trying to create the /mr-history/done directory

[Kevin Buckley]

On 21 July 2017 at 04:04, Erik Krogen  wrote:
> Hi Kevin,
>
> Since you are using the "jhs" keytab with principal "jhs/_h...@realm.tld",
> the JHS is authenticating itself as the jhs user (which is the actual
> important part, rather than the user the process is running as). If you want
> it to be the "mapred" user, you should change the keytab/principal you use
> (mapred.jobhistory.{principal,keytab}).

I'll certainly give that a go Erik, however, the way I read the

>> The hadoop-2.8.0  docs SecureMode page also suggests that one would need to
>> play around with the
>>
>> hadoop.security.auth_to_local

bits suggested to me that if you set things up such that

===
$ hadoop org.apache.hadoop.security.HadoopKerberosName
jhs/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz
17/07/20 17:42:50 INFO util.KerberosName: Non-simple name
mapred/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz after auth_to_local rule
RULE:[2:$1/$2@$0](jhs/.*)s/jhs/mapred/
Name: jhs/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz to
mapred/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz


(or even used a rule that just mapped the principal to a simple "mapred"
because I tried that too !) told you it was remapping the user, then it would
remap for all instances of the user, within the Hadoop instance..

Let's see.
Cheers again for the feedback.

-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

Re: Kerberised JobHistory Server not starting: User jhs trying to create the /mr-history/done directory

[Kevin Buckley]

On 21 July 2017 at 13:25, Kevin Buckley
 wrote:
> On 21 July 2017 at 04:04, Erik Krogen  wrote:
>> Hi Kevin,
>>
>> Since you are using the "jhs" keytab with principal "jhs/_h...@realm.tld",
>> the JHS is authenticating itself as the jhs user (which is the actual
>> important part, rather than the user the process is running as). If you want
>> it to be the "mapred" user, you should change the keytab/principal you use
>> (mapred.jobhistory.{principal,keytab}).
>
> I'll certainly give that a go Erik, however, the way I read the
>
>>> The hadoop-2.8.0  docs SecureMode page also suggests that one would need to
>>> play around with the
>>>
>>> hadoop.security.auth_to_local
>
> bits suggested to me that if you set things up such that
>
> ===
> $ hadoop org.apache.hadoop.security.HadoopKerberosName
> jhs/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz
> 17/07/20 17:42:50 INFO util.KerberosName: Non-simple name
> mapred/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz after auth_to_local rule
> RULE:[2:$1/$2@$0](jhs/.*)s/jhs/mapred/
> Name: jhs/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz to
> mapred/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz
> 
>
> (or even used a rule that just mapped the principal to a simple "mapred"
> because I tried that too !) told you it was remapping the user, then it would
> remap for all instances of the user, within the Hadoop instance..
>
> Let's see.

OK,

so it would appear that despite the Hadoop docs appearing to suggest that
you only need the three usernames, 'hdfs', 'yarn' and 'mapred'. if you do use
the principal from the docs, which has the jhs component, then even if you
do try to map users using 'hadoop.security.auth_to_local', your JobHistory
server will start up, inside Hadoop running as a 'jhs' user.

That would seem to be a bit of a trap for the unaware/unwary that the docs
could easily improve upon ?

Thanks again for the pointer to the correct interpreation of the docs, Erik,
Kevin

-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

Re: Kerberised JobHistory Server not starting: User jhs trying to create the /mr-history/done directory

[Kevin Buckley]

On 25 July 2017 at 03:21, Erik Krogen  wrote:
> Hey Kevin,
>
> Sorry, I missed your point about using auth_to_local. You're right that you 
> should be able to use that for what you're trying to achieve. I think it's 
> just that your rule is wrong; I believe it should be:
>
> RULE:[2:$1@$0](jhs/.*@ECS.VUW.AC.NZ)s/.*/mapred/
>
> HTH

If I ever get a chance to retrace my steps, Erik, then I'll give it a go.

As things stand I went with the altered username in the keytab and
things seem to be working, although I have one other issue that I'm
about to start a new thread for.

Cheers again,
Kevin

-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

Hadoop 2.8.0: Use of container-executor.cfg to restrict access to MapReduce jobs

[Kevin Buckley]

Hi again

early on in my attempts to Kerberise our Hadoop instance, I had seen an
error message that suggested I needed to add a list of users who could
run jobs into the last line of Hadoop's

container-executor.cfg

for which the default content is

yarn.nodemanager.linux-container-executor.group=#configured value of
yarn.nodemanager.linux-container-executor.group
banned.users=#comma separated list of users who can not run applications
min.user.id=1000#Prevent other super-users
allowed.system.users=##comma separated list of system users who CAN
run applications


and after I had dropped the min.user.id to allow for the yarn user in
our systems to run jobs AND added a list of users higher than that,
those other users were able to run jobs.

I now came to test out removing a user from the "allowed" list and I
can't seem to prevent that user from running MapReduce jobs, no
matter which of the various daemons I stop and start, including
shutting down and restarting the whole thing.

Should I be reading that

allowed.system.users=

list to be a list of UIDs from BELOW the

min.user.id=

list, rather than an actual "only allow users in the list" to run jobs list ?

Clealry, one can't run jobs if one doesn't have access to directories
to put data into, so that's a kind of "job control" ACL of itself but I
was hoping that the underlying HDFS might contain a wider set of
users than those allowed to run jobs at any given time, in which case,
altering the ability via the

container-executor.cfg

list seemed a simple way to achieve that.

Any clues/insight welcome,
Kevin

---
Kevin M. Buckley

eScience Consultant
School of Engineering and Computer Science
Victoria University of Wellington
New Zealand

-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

Hadoop 2.8.0: Job console output suggesting non-existent rmserver 8088:proxy URI

[Kevin Buckley]

Hi again,

my attempts to Kerberise our Hadoop instance seem to
have things working OK, although one of the users has
reported the following issue:

The console output, from a running job, suggests following a link to
the RM server's WebGUI, akin to

  http://rmserver.ecs.vuw.ac.nz:8088/proxy/application_1234567890123_4567/

but that URI doesn't appear to be being served by the RM server's WebGUI.


However, starting from the RM server's WebGUI "About the Cluster"
page's Applications view:

  http://rmserver.ecs.vuw.ac.nz:8088/cluster/apps

we can get to the following page, related to the job

  http://rmserver.ecs.vuw.ac.nz:8088/cluster/app/application_1234567890123_4567/


Similarly, over on the Job History Server, we can get to a page,
related to the job

 http://jhserver.ecs.vuw.ac.nz:19888/jobhistory/job/job_1234567890123_4567/



What, if anything, is likely to be missing from the configuration,
that produces the "8088/proxy" path in the URIs that the console
output presents ?

Kevin

---
Kevin M. Buckley

eScience Consultant
School of Engineering and Computer Science
Victoria University of Wellington
New Zealand

-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

Re: Hadoop 2.8.0: Job console output suggesting non-existent rmserver 8088:proxy URI

[Kevin Buckley]

On 9 September 2017 at 05:17, Ravi Prakash  wrote:

> I'm not sure my reply will be entirely helpful, but here goes.

It sheds more light on things than I previously understood, Ravi, so cheers

> The ResourceManager either proxies your request to the ApplicationMaster (if
> the application is running), or (once the application is finished) serves it
> itself if the job is in the "cache" (usually the last 1 applications) or
> redirects to the MapReduce JHS if its a MapReduce job.

That suggests that I don't have the ApplicationMaster(s) setup correctly
(or possibly at all!) or that the caching setup is wrong because the
ResourceManager and JobHistoryServer clearly both has the Job Info
once the Jobs have ended, as I listed before

>  
> http://rmserver.ecs.vuw.ac.nz:8088/cluster/app/application_1234567890123_4567/
>
> Similarly, over on the Job History Server, we can get to a page,
> related to the job
>
>  http://jhserver.ecs.vuw.ac.nz:19888/jobhistory/job/job_1234567890123_4567/

however, trying to access those, though the "proxy channel"

http://rmserver.ecs.vuw.ac.nz:8088/proxy/

URI doesn't take anyone anywhere.

Thanks again for the insight: I think know where I need to look now,
Kevin

-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

Can start two RM in one cluster

[kevin su]

Hi all,

There is a way to start two Namenode, and request for both of them

So can we start two RM as well ?

Another question is every times if we want to allocate resource from RM, it
will find which node manager has resources.

Is RM send every request to all node manager, and get all NM report, or
there is a meta-data in RM, so it will directly find one of NM to get
resources ?

Thanks in advanced

Kevin
Best regards

yarn launch docker error

[kevin su]

Hi all,

I follow
https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-site/DockerContainers.html#Cluster_Configuration

,and try to launch applications using docker container, but I got this error

I use * hadoop-3.2.0* with *library/openjdk:8* docker image

Exception message: Launch container failed
Shell error output: Configuration does not allow docker mount
'/opt/hadoop-3.2.0/logs/userlogs/application_1560115840519_0003/container_1560115840519_0003_01_02:/opt/hadoop-3.2.0/logs/userlogs/application_1560115840519_0003/container_1560115840519_0003_01_02:rw',
realpath=/opt/hadoop-3.2.0/logs/userlogs/application_1560115840519_0003/container_1560115840519_0003_01_02
Error constructing docker command, docker error code=14, error
message='Invalid docker read-write mount'

Shell output: main : command provided 4main : run as user is kevinmain
: requested yarn user is kevin
Creating script paths...
Creating local dirs...

[2019-06-10 05:32:42.367]Container exited with a non-zero exit code
29. [2019-06-10 05:32:42.367]Container exited with a non-zero exit
code 29.


my container-executor.cfg

1 yarn.nodemanager.linux-container-executor.group=kevin
 2 banned.users=*#*4 allowed.system.users=#5
feature.tc.enabled=false6 7 8 [docker]9   module.enabled=true10
docker.privileged-containers.enabled=false
11   
docker.allowed.capabilities=SYS_CHROOT,MKNOD,SETFCAP,SETPCAP,FSETID,CHOWN,AUDIT_W
   RITE,SETGID,NET_RAW,FOWNER,SETUID,DAC_OVERRIDE,KILL,NET_BIND_SERVICE12
  docker.allowed.networks=bridge,host,none13
docker.trusted.registries=library,openjdk14
docker.allow.ro-mounts=/usr/local/hadoop,/etc/passwd,/etc/group15
docker.allow.rw-mounts=/opt/hadoop-3.2.0/logs


what error in my config

Thanks in advanced


Kevin,

Best Regards

Dose Yarn shared cache use memory ?

[kevin su]

Hi all,

I want make sure that yarn shared cache and distributed cache whether used
memory or not.

I didn’t find the implementation that store data in memory, and it just
store data in hdfs(disk)

If it didn’t use memory, dose it have any chance to change Yarn shared
cache, so it can store specific data in memory, maybe it could let
Mapreduce much faster.

Thanks in advanced

BR,
Kevin Su