Re: What is the class that launches the reducers?

[Hitesh Shah]

Have you considered trying to use Tez with a 3-vertex DAG instead of trying to 
change the MR framework? i.e. A->B, A->C, B->C where A is the original map, C 
is the reducer and B being the verification stage I assume and C is configured 
to not start doing any work until B’s verification completes? The above may or 
may not fit your requirements. Feel free to drop any questions you have to the 
user@tez mailing list if you think about going down this path. 

thanks
— Hitesh


> On Aug 25, 2016, at 11:07 PM, xeon Mailinglist  
> wrote:
> 
> Right now the map and reduce task produces digests of the output. This
> logic is inside the map and reduce functions. I need to pause the execution
> when all maps finish because there will be an external program that is
> synchronizing several mapreduce runtimes. When all map tasks finish from
> the several jobs, the map output will be verified. Then, this external
> program will resume the execution.
> 
> I really want to create a knob in mapreduce by modifying the source code,
> because with this knob I can exclude the identity maps execution and boost
> the performance. I think the devs should create this feature.
> 
> Anyway, I am looking in the source code for the part where reduce tasks are
> set to launch. Does anyone know which class launches the reduce tasks in
> mapreduce v2?
> 
> On Aug 26, 2016 02:07, "Daniel Templeton"  wrote:
> 
>> How are you intending to verify the map output?  It's only partially
>> dumped to disk.  None of the intermediate data goes into HDFS.
>> 
>> Daniel
>> 
>> On Aug 25, 2016 4:10 PM, "xeon Mailinglist" 
>> wrote:
>> 
>>> But then I need to set identity maps to run the reducers. If I suspend a
>>> job after the maps finish, I don't need to set identity maps up. I want to
>>> suspend a job so that I don't run identity maps and get better
>>> performance.
>>> 
>>> On Aug 25, 2016 10:12 PM, "Haibo Chen"  wrote:
>>> 
>>> One thing you can try is to write a map-only job first and then verify the
>>> map out.
>>> 
>>> On Thu, Aug 25, 2016 at 1:18 PM, xeon Mailinglist <
>>> xeonmailingl...@gmail.com
 wrote:
>>> 
 I am using Mapreduce v2.
 
 On Aug 25, 2016 8:18 PM, "xeon Mailinglist" 
 wrote:
 
> I am trying to implement a mechanism in MapReduce v2 that allows to
> suspend and resume a job. I must suspend a job when all the mappers
 finish,
> and resume the job from that point after some time. I do this,
>>> because I
> want to verify the integrity of the map output before executing the
> reducers.
> 
> I am looking for the class that tells when the Reduce tasks should
>>> start.
> Does anyone know where is this?
> 
 
>>> 
>> 


-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

Re: how to use Yarn API to find task/attempt status

[Hitesh Shah]

You would use YARN apis as mentioned my David. Look for “PendingMB” from 
“RM:8088/jmx” to see allocated/reserved/pending stats on a per queue basis. 
There is probably a WS that exposes similar data. At the app level, something 
like "http://RM:8088/ws/v1/cluster/apps/application_1457573549805_0001” will 
give you details (only for a running app ) on allocated MB, running containers 
and pending resourceRequests, clusterUsagePercentage, etc. 

thanks
— Hitesh


On Mar 10, 2016, at 9:21 AM, Frank Luo  wrote:

> Thanks David/Jeff.
>  
> To avoid further confusions, let me make sure I am clear on what I am trying 
> to do: I would like to know how many hours in a day my cluster is running at 
> its full capacity, and when that happens, how long is my waiting queue. I 
> founded similar information on Ambari as below, but I’d like to dive deeper, 
> hence asking.
>  
> From what I see, container per job information, especially pending 
> containers, is only available from an application’s trackingUrl, but that 
> just applies to M/R jobs. I am not able to get the same information from a 
> Tez applications’ trackingUrl (Tez’s url doesn’t do anything for hdp2.2).  So 
> how does Ambari find the information out?
>  
>  
> 
>  
> From: David Morel [mailto:dmo...@amakuru.net] 
> Sent: Thursday, March 10, 2016 1:03 AM
> To: Jeff Zhang
> Cc: user@hadoop.apache.org; Frank Luo
> Subject: Re: how to use Yarn API to find task/attempt status
>  
> The REST API should help. A working implementation (in perl, not java, sorry) 
> is visible here : http://search.cpan.org/dist/Net-Hadoop-YARN/
> Read the comments, they matter :-)
> 
> Le 10 mars 2016 7:28 AM, "Jeff Zhang"  a écrit :
> If it is for M/R, then maybe this is what you want 
> https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/mapreduce/JobStatus.html
>  
>  
>  
> On Thu, Mar 10, 2016 at 1:58 PM, Frank Luo  wrote:
> Let’s say there are 10 standard M/R jobs running. How to find how many tasks 
> are done/running/pending?
>  
> From: Jeff Zhang [mailto:zjf...@gmail.com] 
> Sent: Wednesday, March 09, 2016 9:33 PM
> To: Frank Luo
> Cc: user@hadoop.apache.org
> Subject: Re: how to use Yarn API to find task/attempt status
>  
> I don't think it is related with yarn. Yarn don't know about task/task 
> attempt, it only knows containers. So it should be your application to 
> provide such function. 
>  
> On Thu, Mar 10, 2016 at 11:29 AM, Frank Luo  wrote:
> Anyone had a similar issue and knows the answer?
>  
> From: Frank Luo 
> Sent: Wednesday, March 09, 2016 1:59 PM
> To: 'user@hadoop.apache.org'
> Subject: how to use Yarn API to find task/attempt status
>  
> I have a need to programmatically find out how many tasks are pending in 
> Yarn. Is there a way to do it through a Java API?
>  
> I looked at YarnClient, but not able to find what I need.
>  
> Thx in advance.
>  
> Frank Luo
> This email and any attachments transmitted with it are intended for use by 
> the intended recipient(s) only. If you have received this email in error, 
> please notify the sender immediately and then delete it. If you are not the 
> intended recipient, you must not keep, use, disclose, copy or distribute this 
> email without the author’s prior permission. We take precautions to minimize 
> the risk of transmitting software viruses, but we advise you to perform your 
> own virus checks on any attachment to this message. We cannot accept 
> liability for any loss or damage caused by software viruses. The information 
> contained in this communication may be confidential and may be subject to the 
> attorney-client privilege.
> 
> 
> 
>  
> --
> Best Regards
> 
> Jeff Zhang
> This email and any attachments transmitted with it are intended for use by 
> the intended recipient(s) only. If you have received this email in error, 
> please notify the sender immediately and then delete it. If you are not the 
> intended recipient, you must not keep, use, disclose, copy or distribute this 
> email without the author’s prior permission. We take precautions to minimize 
> the risk of transmitting software viruses, but we advise you to perform your 
> own virus checks on any attachment to this message. We cannot accept 
> liability for any loss or damage caused by software viruses. The information 
> contained in this communication may be confidential and may be subject to the 
> attorney-client privilege.
> 
> 
> 
>  
> --
> Best Regards
> 
> Jeff Zhang
> This email and any attachments transmitted with it are intended for use by 
> the intended recipient(s) only. If you have received this email in error, 
> please notify the sender immediately and then delete it. If you are not the 
> intended recipient, you must not keep, use, disclose, copy or distribute this 
> email without the author’s prior permission. We take precautions to minimize 
> the risk of transmitting software viruses, but we advise you to 

Re: Can't run hadoop examples with YARN Single node cluster

[Hitesh Shah]

+common-user 

On Mar 7, 2016, at 3:42 PM, Hitesh Shah <hit...@apache.org> wrote:

> 
> On Mar 7, 2016, at 1:50 PM, José Luis Larroque <larroques...@gmail.com> wrote:
> 
>> Hi again guys, i could, finally, find what the issue was!!!
>> 
>> 
> 
>> 
>> mapreduce.map.java.opts
>> 256
>> 
>> 
>> 
>> mapreduce.reduce.java.opts
>> 256
>> 
>> 
>> 
>> If i suppress the last two properties ( mapreduce.map.java.opts , 
>> mapreduce.reduce.java.opts ), wordcount works!
>> 
> 
> That is because the java opts values are invalid. The value should be 
> -Xmx256m for both values instead of just 256. 
> 
> thanks
> -- Hitesh

Re: Yarn app: Cannot run "java -jar" container

[Hitesh Shah]

Ideally, the “yarn logs -application” command should give you the logs for the 
container in question and the stdout/stderr there usually gives you a good 
indication on what is going wrong. 

Second more complex option: 
   - Set yarn.nodemanager.delete.debug-delay-sec to say 1200 or a large enough 
value. Restart all NMs.
   - Run your application. 
   - Find the node on which your container failed. 
   - Search through the yarn nodemanager local-dirs to find the 
launch_container.sh for your container. 
   - Look at its contents to see if things are being setup correctly. Run it 
manually to debug. 

— Hitesh

On Jan 22, 2016, at 5:52 AM, Kristoffer Sjögren  wrote:

> Hi
> 
> I'm trying to run a yarn 2.7.1 application using a basic boilerplate
> [1]. But I have trouble running the container with an executable jar
> file using the following args list.
> 
>List arg = Collections.singletonList(
>  "/usr/jdk64/jdk1.8.0_40/bin/java -jar app-1.0.0-SNAPSHOT.jar" +
>" 1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout" +
>" 2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr"
>);
> 
> I'm not really sure why it doesn't work because the diagnostics
> message on tells me exit code 1 (see below). Neither the stdout or
> stderr files appear, and I have tried to pipe them to /tmp/stdout etc.
> 
> How do I debug this error? Is the diagnostics message the only way? I
> have tried a gazillion different combinations of running the
> container, and the process is very time consuming and frustrating when
> there isn't any information to debug.
> 
> Any tips or pointers on how to trace this error down?
> 
> Cheers,
> -Kristoffer
> 
> [1] 
> https://github.com/hortonworks/simple-yarn-app/tree/master/src/main/java/com/hortonworks/simpleyarnapp
> 
> 
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)
> at org.apache.hadoop.util.Shell.run(Shell.java:487)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
> at 
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 
> 
> Container exited with a non-zero exit code 1
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: user-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

Re: Should AMRMClientAsync#CallbackHandler add method onAMCommand ?

[Hitesh Shah]

Please look at CallbackHandler::onShutdownRequest()

thanks
— Hitesh 

On Aug 13, 2015, at 6:55 AM, Jeff Zhang zjf...@gmail.com wrote:

 I see that AllocateResponse has AMCommand which may request AM to resync or 
 shutdown, but I don't see AMRMClientAsync#CallbackHandler has any method to 
 handle that.   Should AMRMClientAsync#CallbackHandler add method onAMCommand ?
 
 -- 
 Best Regards
 
 Jeff Zhang

Re: node location of mapreduce task from CLI

[Hitesh Shah]

Maybe try the web services for the MR AM: 
https://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html
 ?

— Hitesh 

On Jul 13, 2015, at 3:17 PM, Tomas Delvechio tomasdelvechi...@gmail.com wrote:

 Hi for all, 
 
 I'm trying to get the location of scheduled mapreduce tasks for a job in my 
 cluster from CLI because I don't have access to WebUI.
 
 I saw [1] and [2], but I'm not lucky.
 
 For location of reducer, I want to get some like this:
 
 TASK_ID
 TYPE (MAP or REDUCE)
 IP_SLAVE
 
 Any clue will be fine.
 
 Thanks in advance.
 
 [1]: 
 https://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html
 [2]: 
 https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/YarnCommands.html
 
 
 -- 
 Tomas Delvechio
 

Re: timelineclient in 2.4.0.2.1.11.0-891

[Hitesh Shah]

The error seems to clearly indicate that you are submitting to an invalid queue:
java.io.IOException: Failed to run job : Application 
application_1435937105729_0100 submitted by user  to unknown queue: 
default”

You may want to address the queue name issue first before looking into 
enabling/disabling Timeline.

Furthermore, Apache releases are not versioned using 6 digits 
(https://hadoop.apache.org/releases.html). For your deployment and the changes 
across the 2 versions, you may wish to contact the Hadoop distro vendor that 
has been used for your cluster. 

thanks
— Hitesh 

On Jul 10, 2015, at 2:32 AM, Krautwurm, Florian florian.krautw...@siemens.com 
wrote:

 Hi,
  
 for reasons I have to use hadoop 2.4.0.2.1.11.0-891, but now I get several 
 errors regarding  pig jobs, which are failing:
  
 …
 2015-07-08 12:19:30,883 [JobControl] INFO  
 org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob - 
 PigLatin:sthToHive.pig got an error while submitting
 java.io.IOException: Failed to run job : Application 
 application_1435937105729_0100 submitted by user  to unknown queue: 
 default
 at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:300)
 at 
 org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
 at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
 at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
 at 
 org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at 
 org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
 at 
 org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
 at java.lang.Thread.run(Thread.java:745)
 at 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:271)
 …
  
  
 I am now using HadoopVersion 2.4.0.2.1.11.0-891 and PigVersion 
 0.12.1.2.1.11.0-891.
  
  
 Everything was working as expected with HadoopVersion 2.4.0.2.1.2.0-402 and 
 PigVersion 0.12.1.2.1.2.0-402.
  
 The only difference in my logfiles are the outputs of the TimelineClient and 
 I also found several issues (but not mine) with it online.
  
 I also know, that there was a patch in YARN-1962, which disables the 
 timelineserver by default.
 Unfortunately I can only find release notes for version 2.4.0 and 2.4.1 but 
 none of the minor releases in between.
  
 To convince my IT to maybe disable the timelineserver, I need to know if  
 2.4.0.2.1.11.0-891 is a Version, where the server is included in hadoop 2.4.0 
 but not disabled by default (as it seems to be not included in 
 2.4.0.2.1.2.0-402).
  
 Where can I get such information?
  
 Thanks an best regards,
 Florian

Re: run yarn container as specific user

[Hitesh Shah]

Is you app code running within the container also being run within a UGI.doAs() 
? 

You can use the following in your code to create a UGI for the “actual” user 
and run all the logic within that: 

code
actualUserUGI = UserGroupInformation.createRemoteUser(System
.getenv(ApplicationConstants.Environment.USER.toString()));
/code

Your other option is to try and get the LinuxContainerExecutor working on a 
non-secure cluster ( not sure if that is trivial to do ).

— Hitesh

On Dec 11, 2014, at 12:04 PM, Tim Williams william...@gmail.com wrote:

 I'm able to use the UGI.doAs(..) to launch a yarn app and, through the
 ResourceManager, both the ApplicationMaster and Containers are
 associated with the correct user.  But the process on the node itself
 really runs as the yarn user.  The problem is that the yarn app writes
 data to DFS and its being written as yarn, since that's what the real
 process is.  This is an non-secure cluster.  I've yet to stumble upon
 a solution that doesn't feel icky.  What's the right way to achieve
 this?
 
 Thanks,
 --tim

Re: building Apache Hadoop (main, Tez)

[Hitesh Shah]

Hi Alexey, 

Would you mind sharing details on the issues that you are facing? 

For both hadoop and tez, refer to the respective BUILDING.txt as it contains 
some basic information on required tools to build the project ( maven, protoc, 
etc ).

For hadoop, you should just need to run “mvn install -DskipTests” at the very 
top of the source tree. 
For tez, modify the top-level pom.xml to set hadoop.version to the version of 
hadoop that you just compiled ( from the previous step ) and then run “mvn 
install -DskipTests”

— Hitesh

On Nov 22, 2014, at 10:45 AM, Yakubovich, Alexey 
alexey.yakubov...@searshc.com wrote:

 Hi,
 
 I am trying to learn how to build  couple of projects Apache Hadoop 
 distribution, particularly:
 
 1. The Hadoop-min project 
 
 https://github.com/apache/hadoop  = hadoop-mapreduce-projct  = 
 hadoop-mapreduce-examples
 groupID
 org.apache.hadoop
 artifactID
 hadoop-main
 
 2. Tez project
 
 http://tez.apache.org/install_pre_0_5_0.html - tez 
 groupID
 org.apache.tez 
 artifactID
 tez
 
 Now, I spent quite some time, no success yet.
 Is ip possible at all, of you suppose to use Cloudera (5) or HDP (2.1)?
 
 It seems that you should keep a particular order in building sub-modules ….
 
 Thank you for any help
 
 Alexey
 This message, including any attachments, is the property of Sears Holdings 
 Corporation and/or one of its subsidiaries. It is confidential and may 
 contain proprietary or legally privileged information. If you are not the 
 intended recipient, please delete it without reading the contents. Thank you.

Re: in MR on YARN, can I change my applicationMaster code?

[Hitesh Shah]

Have you considered https://issues.apache.org/jira/browse/MAPREDUCE-4421 ? 

— Hitesh

On Nov 6, 2014, at 4:09 PM, Yang tedd...@gmail.com wrote:

 we are hit with this bug https://issues.apache.org/jira/browse/YARN-2175  
 
 I could either change the NodeManager, or ApplicationMaster, but NM is part 
 of the infrastructure installation and owned by our company ops, so I can't 
 change that; 
 let's say I modify the ApplicationMaster code and include that with my fat 
 jar for my hadoop job,
 would the launched job running my own AM code or the system hadoop's AM ?
 
 
 thanks
 Yang

Re: Hadoop 2.0 job simulation using MiniCluster

[Hitesh Shah]

Maybe check TestMRJobs.java ( 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
 ) ? 

— Hitesh 

On Oct 21, 2014, at 10:01 AM, Yehia Elshater y.z.elsha...@gmail.com wrote:

 Hi All,
 
 I am wondering how can I simulate a Hadoop 2.0 Job using MiniCluster and 
 MiniDFS ? for example, I want to simulate a submission of a Hadoop job with 
 some map/reduce tasks given some data input splits.
 
 Your help is really appreciated.
 Yehia

Re: ResourceManager shutting down

[Hitesh Shah]

Hi John

Would you mind filing a jira with more details. The RM going down just because 
a host was not resolvable or DNS timed out is something that should be 
addressed.

thanks
-- Hitesh

On Mar 13, 2014, at 2:29 PM, John Lilley wrote:

 Never mind… we figured out its DNS entry was going missing.
 john
  
 From: John Lilley [mailto:john.lil...@redpoint.net] 
 Sent: Thursday, March 13, 2014 2:52 PM
 To: user@hadoop.apache.org
 Subject: ResourceManager shutting down
  
 We have this erratic behavior where every so often the RM will shutdown with 
 an UnknownHostException.  The odd thing is, the host it complains about have 
 been in use for days at that point without problem.  Any ideas?
 Thanks,
 John
  
  
 2014-03-13 14:38:14,746 INFO  rmapp.RMAppImpl (RMAppImpl.java:handle(578)) - 
 application_1394204725813_0220 State change from ACCEPTED to RUNNING
 2014-03-13 14:38:15,794 FATAL resourcemanager.ResourceManager 
 (ResourceManager.java:run(449)) - Error in handling event type NODE_UPDATE to 
 the scheduler
 java.lang.IllegalArgumentException: java.net.UnknownHostException: 
 skitzo.office.datalever.com
 at 
 org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:418)
 at 
 org.apache.hadoop.yarn.server.utils.BuilderUtils.newContainerToken(BuilderUtils.java:247)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager.createContainerToken(RMContainerTokenSecretManager.java:195)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainerToken(LeafQueue.java:1297)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1345)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1211)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1170)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:871)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:645)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:690)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:734)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:440)
 at java.lang.Thread.run(Thread.java:662)
 Caused by: java.net.UnknownHostException: skitzo.office.datalever.com
 ... 15 more
 2014-03-13 14:38:15,794 INFO  resourcemanager.ResourceManager 
 (ResourceManager.java:run(453)) - Exiting, bbye..
 2014-03-13 14:38:15,911 INFO  mortbay.log (Slf4jLog.java:info(67)) - Stopped 
 selectchannelconnec...@metallica.office.datalever.com:8088
 2014-03-13 14:38:16,013 ERROR delegation.AbstractDelegationTokenSecretManager 
 (AbstractDelegationTokenSecretManager.java:run(557)) - InterruptedExcpetion 
 recieved for ExpiredTokenRemover thread java.lang.InterruptedException: sleep 
 interrupted
 2014-03-13 14:38:16,013 INFO  impl.MetricsSystemImpl 
 (MetricsSystemImpl.java:stop(200)) - Stopping ResourceManager metrics 
 system...
 2014-03-13 14:38:16,014 INFO  impl.MetricsSystemImpl 
 (MetricsSystemImpl.java:stop(206)) - ResourceManager metrics system stopped.
 2014-03-13 14:38:16,014 INFO  impl.MetricsSystemImpl 
 (MetricsSystemImpl.java:shutdown(572)) - ResourceManager metrics system 
 shutdown complete.
 2014-03-13 14:38:16,015 WARN  amlauncher.ApplicationMasterLauncher 
 (ApplicationMasterLauncher.java:run(98)) - 
 org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
  interrupted. Returning.
 2014-03-13 14:38:16,015 INFO  ipc.Server (Server.java:stop(2442)) - Stopping 
 server on 8141
 2014-03-13 14:38:16,017 INFO  ipc.Server (Server.java:stop(2442)) - Stopping 
 server on 8050
 … and so on, it shuts down
  

Re: ResourceManager shutting down

[Hitesh Shah]

Hi John

Would you mind filing a jira with more details. The RM going down just because 
a host was not resolvable or DNS timed out is something that should be 
addressed.

thanks
-- Hitesh

On Mar 13, 2014, at 2:29 PM, John Lilley wrote:

 Never mind… we figured out its DNS entry was going missing.
 john
  
 From: John Lilley [mailto:john.lil...@redpoint.net] 
 Sent: Thursday, March 13, 2014 2:52 PM
 To: user@hadoop.apache.org
 Subject: ResourceManager shutting down
  
 We have this erratic behavior where every so often the RM will shutdown with 
 an UnknownHostException.  The odd thing is, the host it complains about have 
 been in use for days at that point without problem.  Any ideas?
 Thanks,
 John
  
  
 2014-03-13 14:38:14,746 INFO  rmapp.RMAppImpl (RMAppImpl.java:handle(578)) - 
 application_1394204725813_0220 State change from ACCEPTED to RUNNING
 2014-03-13 14:38:15,794 FATAL resourcemanager.ResourceManager 
 (ResourceManager.java:run(449)) - Error in handling event type NODE_UPDATE to 
 the scheduler
 java.lang.IllegalArgumentException: java.net.UnknownHostException: 
 skitzo.office.datalever.com
 at 
 org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:418)
 at 
 org.apache.hadoop.yarn.server.utils.BuilderUtils.newContainerToken(BuilderUtils.java:247)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager.createContainerToken(RMContainerTokenSecretManager.java:195)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainerToken(LeafQueue.java:1297)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1345)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1211)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1170)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:871)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:645)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:690)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:734)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:440)
 at java.lang.Thread.run(Thread.java:662)
 Caused by: java.net.UnknownHostException: skitzo.office.datalever.com
 ... 15 more
 2014-03-13 14:38:15,794 INFO  resourcemanager.ResourceManager 
 (ResourceManager.java:run(453)) - Exiting, bbye..
 2014-03-13 14:38:15,911 INFO  mortbay.log (Slf4jLog.java:info(67)) - Stopped 
 selectchannelconnec...@metallica.office.datalever.com:8088
 2014-03-13 14:38:16,013 ERROR delegation.AbstractDelegationTokenSecretManager 
 (AbstractDelegationTokenSecretManager.java:run(557)) - InterruptedExcpetion 
 recieved for ExpiredTokenRemover thread java.lang.InterruptedException: sleep 
 interrupted
 2014-03-13 14:38:16,013 INFO  impl.MetricsSystemImpl 
 (MetricsSystemImpl.java:stop(200)) - Stopping ResourceManager metrics 
 system...
 2014-03-13 14:38:16,014 INFO  impl.MetricsSystemImpl 
 (MetricsSystemImpl.java:stop(206)) - ResourceManager metrics system stopped.
 2014-03-13 14:38:16,014 INFO  impl.MetricsSystemImpl 
 (MetricsSystemImpl.java:shutdown(572)) - ResourceManager metrics system 
 shutdown complete.
 2014-03-13 14:38:16,015 WARN  amlauncher.ApplicationMasterLauncher 
 (ApplicationMasterLauncher.java:run(98)) - 
 org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
  interrupted. Returning.
 2014-03-13 14:38:16,015 INFO  ipc.Server (Server.java:stop(2442)) - Stopping 
 server on 8141
 2014-03-13 14:38:16,017 INFO  ipc.Server (Server.java:stop(2442)) - Stopping 
 server on 8050
 … and so on, it shuts down
  

Re: Passing data from Client to AM

[Hitesh Shah]

Adding values to a Configuration object does not really work unless you 
serialize the config into a file and send it over to the AM and containers as a 
local resource. The application code would then need to load in this file using 
Configuration::addResource(). MapReduce does this by taking in all user 
configured values and serializing them in the form of job.xml.

-- Hitesh

On Jan 29, 2014, at 2:42 PM, Jay Vyas wrote:

 while your at it, what about adding values to the Configuration() object, 
 does that still work as a hack for information passing?
 
 
 On Wed, Jan 29, 2014 at 5:25 PM, Arun C Murthy a...@hortonworks.com wrote:
 Command line arguments  env variables are the most direct options.
 
 A more onerous option is to write some data to a file in HDFS, use 
 LocalResource to ship it to the container on each node and get application 
 code to read that file locally. (In MRv1 parlance that is Distributed 
 Cache).
 
 hth,
 Arun
 
 On Jan 29, 2014, at 12:59 PM, Brian C. Huffman bhuff...@etinternational.com 
 wrote:
 
 I'm looking at Distributed Shell as an example for writing a YARN 
 application.
 
 My question is why are the script path and associated metadata saved as 
 environment variables?  Are there any other ways besides environment 
 variables or command line arguments for passing data from the Client to the 
 ApplicationMaster?
 
 Thanks,
 Brian
 
 
 
 
 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/
 
 
 
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to 
 which it is addressed and may contain information that is confidential, 
 privileged and exempt from disclosure under applicable law. If the reader of 
 this message is not the intended recipient, you are hereby notified that any 
 printing, copying, dissemination, distribution, disclosure or forwarding of 
 this communication is strictly prohibited. If you have received this 
 communication in error, please contact the sender immediately and delete it 
 from your system. Thank You.
 
 
 
 -- 
 Jay Vyas
 http://jayunit100.blogspot.com

Re: How to make AM terminate if client crashes?

[Hitesh Shah]

You would probably need to bake this into your own application. By default, a 
client never should need to keep an open active connection with the RM. It 
could keep an active connection with the AM ( application-specific code 
required ) but it would then also have to handle failover to a different AM if 
the first AM crashes and a new attempt is launched by the RM. 

Does your application have access to a zookeeper cluster within your 
installation? It would be simple enough for someone to build a library which 
either uses an ephemeral node or updates some form of an applicationId specific 
keep-alive flag that the AM can monitor to make its decisions. 

@Vinod, on a different note, does it make sense for the RM to run an embedded 
ZK to aid fencing across multiple AM attempts which could then be used by the 
above to piggyback on? 

-- Hitesh

On Jan 13, 2014, at 8:30 AM, Vinod Kumar Vavilapalli wrote:

 The architecture is built around detachable clients. So, no, it doesn't 
 happen automatically. Even if we were to add that feature, it'd be fraught 
 with edge cases - network issues causing app-termination even though client 
 is still alive etc.
 
 Any more details on why this is desired?
 
 +Vinod
 
 On Jan 11, 2014, at 11:37 AM, John Lilley john.lil...@redpoint.net wrote:
 
 We have a YARN application that we want to automatically terminate if the 
 YARN client disconnects or crashes.  Is it possible to configure the 
 YarnClient-RM connection so that if the client terminates the RM 
 automatically terminates the AM?  Or do we need to build our own logic (e.g. 
 a direct client-AM connection) for that?
 Thanks
 John

Re: Unmanaged AMs

[Hitesh Shah]

Hello Kishore, 

An unmanaged AM has no relation to the language being used. An unmanaged AM is 
an AM that is launched outside of the YARN cluster i.e. manually launched 
elsewhere and not by the RM ( using the application submission context provided 
by a client). It was built to be a dev-tool for application developers to be 
able to test their AMs ( attach debuggers, etc ) and is not meant to be 
something that is used in production.

As for other languages, all interactions with the YARN components is via 
protobuf-based RPC and you could use the appropriate language binding for 
protobuf. Take a look at https://github.com/hortonworks/gohadoop - this has 
code for a YARN app written in Go. There is still some work left to get this to 
work seamlessly for all language types but the go code should point you in the 
right direction.

-- Hitesh

On Nov 21, 2013, at 6:18 AM, Krishna Kishore Bonagiri wrote:

 Hi,
 
   I have seen in comments for code in UnmanagedAMLauncher.java that AM can be 
 in any language. What does that mean? Can AM be written in C++ language? If 
 so, how would I be able to be connect to RM and how would I be able to 
 request for containers? I mean what is the interface doing these things? Is 
 there a sample code/example somewhere to get an idea about how to do it?
 
 Thanks,
 Kishore

Re: HDP 2.0 Install fails on repo unavailability

[Hitesh Shah]

BCC'ing user@hadoop.

This is a question for the ambari mailing list. 

-- Hitesh 

On Oct 24, 2013, at 3:36 PM, Jain, Prem wrote:

 Folks,
  
 Trying to install the newly release Hadoop 2.0 using Ambari. I am able to 
 install Ambari, but when I try to install Hadoop 2.0 on rest of the cluster, 
 the installation fails erroring on repo mirror unavailability.  Not sure 
 where I messed up.
  
 Here are the error messages
  
 Output log from AMBARI during Installation
  
 notice: 
 /Stage[1]/Hdp::Snappy::Package/Hdp::Snappy::Package::Ln[32]/Hdp::Exec[hdp::snappy::package::ln
  32]/Exec[hdp::snappy::package::ln 32]/returns: executed successfully
 err: /Stage[main]/Hdp-hadoop/Hdp-hadoop::Package[hadoop]/Hdp::Package[hadoop 
 64]/Hdp::Package::Process_pkg[hadoop 64]/Package[hadoop-libhdfs]/ensure: 
 change from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y 
 install hadoop-libhdfs' returned 1:
  
 Error Downloading Packages:
   hadoop-2.2.0.2.0.6.0-76.el6.x86_64: failure: 
 hadoop/hadoop-2.2.0.2.0.6.0-76.el6.x86_64.rpm from HDP-2.0.6: [Errno 256] No 
 more mirrors to try.
   zookeeper-3.4.5.2.0.6.0-76.el6.noarch: failure: 
 zookeeper/zookeeper-3.4.5.2.0.6.0-76.el6.noarch.rpm from HDP-2.0.6: [Errno 
 256] No more mirrors to try.
   hadoop-hdfs-2.2.0.2.0.6.0-76.el6.x86_64: failure: 
 hadoop/hadoop-hdfs-2.2.0.2.0.6.0-76.el6.x86_64.rpm from HDP-2.0.6: [Errno 
 256] No more mirrors to try.
  
  
 err: /Stage[main]/Hdp-hadoop/Hdp-hadoop::Package[hadoop]/Hdp::Package[hadoop 
 64]/Hdp::Package::Process_pkg[hadoop 64]/Package[hadoop-lzo]/ensure: change 
 from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y 
 install hadoop-lzo' returned 1:
  
 Error Downloading Packages:
   hadoop-2.2.0.2.0.6.0-76.el6.x86_64: failure: 
 hadoop/hadoop-2.2.0.2.0.6.0-76.el6.x86_64.rpm from HDP-2.0.6: [Errno 256] No 
 more mirrors to try.
   zookeeper-3.4.5.2.0.6.0-76.el6.noarch: failure: 
 zookeeper/zookeeper-3.4.5.2.0.6.0-76.el6.noarch.rpm from HDP-2.0.6: [Errno 
 256] No more mirrors to try.
  
  
 notice: 
 /Stage[2]/Hdp-hadoop::Initialize/Configgenerator::Configfile[hdfs-site]/File[/etc/hadoop/conf/hdfs-site.xml]/content:
  content changed '{md5}117224b1cf67c151f8a3d7ac0a157fa5' to 
 '{md5}ba383a94bdde1a0b2eb5c59b1f5b61e7'
 notice: 
 /Stage[2]/Hdp-hadoop::Initialize/Configgenerator::Configfile[capacity-scheduler]/File[/etc/hadoop/conf/capacity-scheduler.xml]/content:
  content changed '{md5}08d7e952b3e2d4fd5a2a880dfcd3a2df' to 
 '{md5}dd3922fc27f72cd78cf2b47f57351b08'
 notice: 
 /Stage[2]/Hdp-hadoop::Initialize/Configgenerator::Configfile[core-site]/File[/etc/hadoop/conf/core-site.xml]/content:
  content changed '{md5}76d06ebce1310be7e65ae0c7e8c3068a' to 
 '{md5}1b626aa016a6f916271f67f3aa22cbbb'
 err: /Stage[main]/Hdp-hadoop/Hdp-hadoop::Package[hadoop]/Hdp::Package[hadoop 
 64]/Hdp::Package::Process_pkg[hadoop 64]/Package[hadoop]/ensure: change from 
 absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install 
 hadoop' returned 1:
  
 Error Downloading Packages:
   hadoop-2.2.0.2.0.6.0-76.el6.x86_64: failure: 
 hadoop/hadoop-2.2.0.2.0.6.0-76.el6.x86_64.rpm from HDP-2.0.6: [Errno 256] No 
 more mirrors to try.
   zookeeper-3.4.5.2.0.6.0-76.el6.noarch: failure: 
 zookeeper/zookeeper-3.4.5.2.0.6.0-76.el6.noarch.rpm from HDP-2.0.6: [Errno 
 256] No more mirrors to try.
  
  
 notice: 
 /Stage[main]/Hdp-hadoop/Hdp-hadoop::Package[hadoop]/Hdp::Package[hadoop 
 64]/Hdp::Package::Process_pkg[hadoop 64]/Anchor[hdp::package::hadoop 
 64::end]: Dependency Package[hadoop-libhdfs] has failures: true
 notice: 
 /Stage[main]/Hdp-hadoop/Hdp-hadoop::Package[hadoop]/Hdp::Package[hadoop 
 64]/Hdp::Package::Process_pkg[hadoop 64]/Anchor[hdp::package::hadoop 
 64::end]: Dependency Package[hadoop-lzo] has failures: true
 notice: 
 /Stage[main]/Hdp-hadoop/Hdp-hadoop::Package[hadoop]/Hdp::Package[hadoop 
 64]/Hdp::Package::Process_pkg[hadoop 64]/Anchor[hdp::package::hadoop 
 64::end]: Dependency Package[hadoop] has failures: true
 notice: 
 /Stage[main]/Hdp-hadoop/Hdp-hadoop::Package[hadoop]/Anchor[hdp-hadoop::package::helper::end]:
  Dependency Package[hadoop-libhdfs] has failures: true
 notice: 
 /Stage[main]/Hdp-hadoop/Hdp-hadoop::Package[hadoop]/Anchor[hdp-hadoop::package::helper::end]:
  Dependency Package[hadoop-lzo] has failures: true
  
 Repo :
  
 http://public-repo-1.hortonworks.com/ambari/centos6/1.x/updates/1.4.1.25
 ErrorCodeNoSuchKey/CodeMessageThe specified key does not 
 exist./MessageKeyambari/centos6/1.x/updates/1.4.1.25/KeyRequestId4693487CE703DB53/RequestIdHostIdB87iAHvcpH7im27HOuEKBJ0F+qPFf+7aXuTe+O7OhLb9WscyxTbV/2yUPXO+KPOJ/HostId/Error
  
  
 Manual install:
  
 [root@dn5 ~]# /usr/bin/yum -d 0 -e 0 -y install hadoop-libhdfs
 Error Downloading Packages:
   hadoop-2.2.0.2.0.6.0-76.el6.x86_64: failure: 
 hadoop/hadoop-2.2.0.2.0.6.0-76.el6.x86_64.rpm from HDP-2.0.6: [Errno 256] No 
 more mirrors to try.
   zookeeper-3.4.5.2.0.6.0-76.el6.noarch: failure: 
 

Re: simple word count program remains un assigned...

[Hitesh Shah]

Hello Gunjan,

This mailing list is for Apache Hadoop related questions. Please post questions 
for other distributions to the appropriate vendor's mailing list.

thanks
-- Hitesh

On Oct 19, 2013, at 11:27 AM, gunjan mishra wrote:

 Hi I am trying to run a simple word count program , like this , job keeps  
 running but is not being assigned to Mapper and reducers ...
 while I checked the status 
 
 ===
 
 [root@localhost ~]# hadoop jar 
 /usr/lib/hadoop-0.20-mapreduce/hadoop-examples-2.0.0-mr1-cdh4.4.0.jar 
 wordcount /usr/read.txt /usr/output
 13/10/19 15:05:02 INFO service.AbstractService: 
 Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
 13/10/19 15:05:02 INFO service.AbstractService: 
 Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
 13/10/19 15:05:03 INFO input.FileInputFormat: Total input paths to process : 1
 13/10/19 15:05:03 INFO mapreduce.JobSubmitter: number of splits:1
 13/10/19 15:05:03 WARN conf.Configuration: mapred.jar is deprecated. Instead, 
 use mapreduce.job.jar
 13/10/19 15:05:03 WARN conf.Configuration: mapred.output.value.class is 
 deprecated. Instead, use mapreduce.job.output.value.class
 13/10/19 15:05:03 WARN conf.Configuration: mapreduce.combine.class is 
 deprecated. Instead, use mapreduce.job.combine.class
 13/10/19 15:05:03 WARN conf.Configuration: mapreduce.map.class is deprecated. 
 Instead, use mapreduce.job.map.class
 13/10/19 15:05:03 WARN conf.Configuration: mapred.job.name is deprecated. 
 Instead, use mapreduce.job.name
 13/10/19 15:05:03 WARN conf.Configuration: mapreduce.reduce.class is 
 deprecated. Instead, use mapreduce.job.reduce.class
 13/10/19 15:05:03 WARN conf.Configuration: mapred.input.dir is deprecated. 
 Instead, use mapreduce.input.fileinputformat.inputdir
 13/10/19 15:05:03 WARN conf.Configuration: mapred.output.dir is deprecated. 
 Instead, use mapreduce.output.fileoutputformat.outputdir
 13/10/19 15:05:03 WARN conf.Configuration: mapred.map.tasks is deprecated. 
 Instead, use mapreduce.job.maps
 13/10/19 15:05:03 WARN conf.Configuration: mapred.output.key.class is 
 deprecated. Instead, use mapreduce.job.output.key.class
 13/10/19 15:05:03 WARN conf.Configuration: mapred.working.dir is deprecated. 
 Instead, use mapreduce.job.working.dir
 13/10/19 15:05:03 INFO mapreduce.JobSubmitter: Submitting tokens for job: 
 job_1382144693199_0005
 13/10/19 15:05:03 INFO client.YarnClientImpl: Submitted application 
 application_1382144693199_0005 to ResourceManager at /0.0.0.0:8032
 13/10/19 15:05:04 INFO mapreduce.Job: The url to track the job: 
 http://localhost.localdomain:8088/proxy/application_1382144693199_0005/
 13/10/19 15:05:04 INFO mapreduce.Job: Running job: job_1382144693199_0005
 
 ===
 
 here is my cloudera distribution 
 
 
 Hadoop 2.0.0-cdh4.4.0
 Subversion 
 file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.4.0/src/hadoop-common-project/hadoop-common
  -r c0eba6cd38c984557e96a16ccd7356b7de835e79
 Compiled by jenkins on Tue Sep  3 19:33:17 PDT 2013
 From source with checksum ac7e170aa709b3ace13dc5f775487180
 This command was run using /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.4.0.jar
 
 
 and the outcome of jps (from root)
 
 -
 [root@localhost ~]# jps
 2202 TaskTracker
 4161 Bootstrap
 3134 DataNode
 3520 Application
 3262 NameNode
 1879 ThriftServer
 1740 Main
 3603 RunJar
 1606 HMaster
 2078 JobTracker
 16277 Jps
 3624 RunJar
 4053 RunJar
 4189 Sqoop
 3582 Bootstrap
 3024 JobHistoryServer
 3379 SecondaryNameNode
 4732 ResourceManager
 
 --
 -- 
 Thanks  Regards
 Gunjan Mishra
 732-200-5839(H)
 917-216-9739(C)

Re: Conflicting dependency versions

[Hitesh Shah]

Hi Albert, 

If you are using distributed cache to push the newer version of the guava jars, 
you can try setting mapreduce.job.user.classpath.first to true. If not, you 
can try overriding the value of mapreduce.application.classpath to ensure that 
the dir where the newer guava jars are present is referenced first in the 
classpath.

-- Hitesh

On Oct 10, 2013, at 3:27 PM, Albert Shau wrote:

 Hi,
 
 I have a yarn application that launches a mapreduce job that has a mapper 
 that uses a newer version of guava than the one hadoop is using.  Because of 
 this, the mapper fails and gets a NoSuchMethod exception.  Is there a way to 
 indicate that application dependencies should be used over hadoop 
 dependencies?
 
 Thanks,
 Albert

Re: Hadoop Yarn

[Hitesh Shah]

Hi Rajesh,

Have you looked at re-using the profiling options to inject the jvm options to 
a defined range of tasks? 
http://hadoop.apache.org/docs/stable/mapred_tutorial.html#Profiling

-- Hitesh

On Aug 29, 2013, at 3:51 PM, Rajesh Jain wrote:

 Hi Vinod
 
 These are jvm parameters to inject agent only on some nodes for sampling. 
 
 Is there a property because code change is not a option. 
 
 Second is there a way to tell the jvms how much data size to process. 
 
 Thanks
 
 Sent from my iPhone
 
 On Aug 29, 2013, at 6:37 PM, Vinod Kumar Vavilapalli vino...@apache.org 
 wrote:
 
 
 You'll have to change the MapReduce code. What options are you exactly 
 looking for and why should they be only applied on some nodes? Some kind of 
 sampling?
 
 More details can help us help you.
 
 Thanks,
 +Vinod Kumar Vavilapalli
 Hortonworks Inc.
 http://hortonworks.com/
 
 On Aug 29, 2013, at 1:59 PM, Rajesh Jain wrote:
 
 I have some jvm options which i want to configure only for a few nodes in 
 the cluster using Hadoop yarn. How do i di it. If i edit the 
 mapred-site.xml it gets applied to all the task jvms. I just want handful 
 of map jvms to have that option and other map jvm not have that options. 
 
 Thanks
 Rajesh
 
 Sent from my iPhone
 
 
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to 
 which it is addressed and may contain information that is confidential, 
 privileged and exempt from disclosure under applicable law. If the reader of 
 this message is not the intended recipient, you are hereby notified that any 
 printing, copying, dissemination, distribution, disclosure or forwarding of 
 this communication is strictly prohibited. If you have received this 
 communication in error, please contact the sender immediately and delete it 
 from your system. Thank You.
 signature.asc

Re: setLocalResources() on ContainerLaunchContext

[Hitesh Shah]

Hi Krishna, 

YARN downloads a specified local resource on the container's node from the url 
specified. In all situtations, the remote url needs to be a fully qualified 
path. To verify that the file at the remote url is still valid, YARN expects 
you to provide the length and last modified timestamp of that file.

If you use an hdfs path such as hdfs://namenode:port/absolute path to file, 
you will need to get the length and timestamp from HDFS. 
If you use file:///, the file should exist on all nodes and all nodes should 
have the file with the same length and timestamp for localization to work. ( 
For a single node setup, this works but tougher to get right on a multi-node 
setup - deploying the file via a rpm should likely work).

-- Hitesh

On Aug 6, 2013, at 11:11 AM, Omkar Joshi wrote:

 Hi,
 
 You need to match the timestamp. Probably get the timestamp locally before 
 adding it. This is explicitly done to ensure that file is not updated after 
 user makes the call to avoid possible errors.
 
 
 Thanks,
 Omkar Joshi
 Hortonworks Inc.
 
 
 On Tue, Aug 6, 2013 at 5:25 AM, Krishna Kishore Bonagiri 
 write2kish...@gmail.com wrote:
 I tried the following and it works!
 String shellScriptPath = file:///home_/dsadm/kishore/kk.ksh;
 
 But now getting a timestamp error like below, when I passed 0 to 
 setTimestamp()
 
 13/08/06 08:23:48 INFO ApplicationMaster: Got container status for 
 containerID= container_1375784329048_0017_01_02, state=COMPLETE, 
 exitStatus=-1000, diagnostics=Resource file:/home_/dsadm/kishore/kk.ksh 
 changed on src filesystem (expected 0, was 136758058
 
 
 
 
 On Tue, Aug 6, 2013 at 5:24 PM, Harsh J ha...@cloudera.com wrote:
 Can you try passing a fully qualified local path? That is, including the 
 file:/ scheme
 
 On Aug 6, 2013 4:05 PM, Krishna Kishore Bonagiri write2kish...@gmail.com 
 wrote:
 Hi Harsh,
The setResource() call on LocalResource() is expecting an argument of type 
 org.apache.hadoop.yarn.api.records.URL which is converted from a string in 
 the form of URI. This happens in the following call of Distributed Shell 
 example, 
 
 shellRsrc.setResource(ConverterUtils.getYarnUrlFromURI(new URI( 
 shellScriptPath)));
 
 So, if I give a local file I get a parsing error like below, which is when I 
 changed it to an HDFS file thinking that it should be given like that only. 
 Could you please give an example of how else it could be used, using a local 
 file as you are saying?
 
 2013-08-06 06:23:12,942 WARN 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
  Failed to parse resource-request
 java.net.URISyntaxException: Expected scheme name at index 0: 
 :///home_/dsadm/kishore/kk.ksh
 at java.net.URI$Parser.fail(URI.java:2820)
 at java.net.URI$Parser.failExpecting(URI.java:2826)
 at java.net.URI$Parser.parse(URI.java:3015)
 at java.net.URI.init(URI.java:747)
 at 
 org.apache.hadoop.yarn.util.ConverterUtils.getPathFromYarnURL(ConverterUtils.java:77)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourceRequest.init(LocalResourceRequest.java:46)
 
 
 
 On Tue, Aug 6, 2013 at 3:36 PM, Harsh J ha...@cloudera.com wrote:
 To be honest, I've never tried loading a HDFS file onto the
 LocalResource this way. I usually just pass a local file and that
 works just fine. There may be something in the URI transformation
 possibly breaking a HDFS source, but try passing a local file - does
 that fail too? The Shell example uses a local file.
 
 On Tue, Aug 6, 2013 at 10:54 AM, Krishna Kishore Bonagiri
 write2kish...@gmail.com wrote:
  Hi Harsh,
 
Please see if this is useful, I got a stack trace after the error has
  occurred
 
  2013-08-06 00:55:30,559 INFO
  org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set
  to /tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
  =
  file:/tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
  2013-08-06 00:55:31,017 ERROR
  org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
  as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
  exist: hdfs://isredeng/kishore/kk.ksh
  2013-08-06 00:55:31,029 INFO
  org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
  DEBUG: FAILED { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null }, File does
  not exist: hdfs://isredeng/kishore/kk.ksh
  2013-08-06 00:55:31,031 INFO
  org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
  Resource hdfs://isredeng/kishore/kk.ksh transitioned from DOWNLOADING to
  FAILED
  2013-08-06 00:55:31,034 INFO
  org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
  Container container_1375716148174_0004_01_02 transitioned from
  LOCALIZING to LOCALIZATION_FAILED
  2013-08-06 00:55:31,035 INFO
  

Re: setLocalResources() on ContainerLaunchContext

[Hitesh Shah]

@Krishna, your logs showed the file error for hdfs://isredeng/kishore/kk.ksh 

I am assuming you have tried dfs -ls /kishore/kk.ksh and confirmed that the 
file exists? Also the qualified path seems to be missing the namenode port. I 
need to go back and check if a path without the port works by assuming the 
default namenode port.

@Harsh, adding a helper function seems like a good idea. Let me file a jira to 
have the above added to one of the helper/client libraries.

thanks
-- Hitesh

On Aug 6, 2013, at 6:47 PM, Harsh J wrote:

 It is kinda unnecessary to be asking developers to load in timestamps and
 length themselves. Why not provide a java.io.File, or perhaps a Path
 accepting API, that gets it automatically on their behalf using the
 FileSystem API internally?
 
 P.s. A HDFS file gave him a FNF, while a Local file gave him a proper
 TS/Len error. I'm guessing there's a bug here w.r.t. handling HDFS
 paths.
 
 On Wed, Aug 7, 2013 at 12:35 AM, Hitesh Shah hit...@apache.org wrote:
 Hi Krishna,
 
 YARN downloads a specified local resource on the container's node from the 
 url specified. In all situtations, the remote url needs to be a fully 
 qualified path. To verify that the file at the remote url is still valid, 
 YARN expects you to provide the length and last modified timestamp of that 
 file.
 
 If you use an hdfs path such as hdfs://namenode:port/absolute path to 
 file, you will need to get the length and timestamp from HDFS.
 If you use file:///, the file should exist on all nodes and all nodes should 
 have the file with the same length and timestamp for localization to work. ( 
 For a single node setup, this works but tougher to get right on a multi-node 
 setup - deploying the file via a rpm should likely work).
 
 -- Hitesh
 
 On Aug 6, 2013, at 11:11 AM, Omkar Joshi wrote:
 
 Hi,
 
 You need to match the timestamp. Probably get the timestamp locally before 
 adding it. This is explicitly done to ensure that file is not updated after 
 user makes the call to avoid possible errors.
 
 
 Thanks,
 Omkar Joshi
 Hortonworks Inc.
 
 
 On Tue, Aug 6, 2013 at 5:25 AM, Krishna Kishore Bonagiri 
 write2kish...@gmail.com wrote:
 I tried the following and it works!
 String shellScriptPath = file:///home_/dsadm/kishore/kk.ksh;
 
 But now getting a timestamp error like below, when I passed 0 to 
 setTimestamp()
 
 13/08/06 08:23:48 INFO ApplicationMaster: Got container status for 
 containerID= container_1375784329048_0017_01_02, state=COMPLETE, 
 exitStatus=-1000, diagnostics=Resource file:/home_/dsadm/kishore/kk.ksh 
 changed on src filesystem (expected 0, was 136758058
 
 
 
 
 On Tue, Aug 6, 2013 at 5:24 PM, Harsh J ha...@cloudera.com wrote:
 Can you try passing a fully qualified local path? That is, including the 
 file:/ scheme
 
 On Aug 6, 2013 4:05 PM, Krishna Kishore Bonagiri 
 write2kish...@gmail.com wrote:
 Hi Harsh,
   The setResource() call on LocalResource() is expecting an argument of 
 type org.apache.hadoop.yarn.api.records.URL which is converted from a 
 string in the form of URI. This happens in the following call of 
 Distributed Shell example,
 
 shellRsrc.setResource(ConverterUtils.getYarnUrlFromURI(new URI( 
 shellScriptPath)));
 
 So, if I give a local file I get a parsing error like below, which is when 
 I changed it to an HDFS file thinking that it should be given like that 
 only. Could you please give an example of how else it could be used, using 
 a local file as you are saying?
 
 2013-08-06 06:23:12,942 WARN 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
  Failed to parse resource-request
 java.net.URISyntaxException: Expected scheme name at index 0: 
 :///home_/dsadm/kishore/kk.ksh
at java.net.URI$Parser.fail(URI.java:2820)
at java.net.URI$Parser.failExpecting(URI.java:2826)
at java.net.URI$Parser.parse(URI.java:3015)
at java.net.URI.init(URI.java:747)
at 
 org.apache.hadoop.yarn.util.ConverterUtils.getPathFromYarnURL(ConverterUtils.java:77)
at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourceRequest.init(LocalResourceRequest.java:46)
 
 
 
 On Tue, Aug 6, 2013 at 3:36 PM, Harsh J ha...@cloudera.com wrote:
 To be honest, I've never tried loading a HDFS file onto the
 LocalResource this way. I usually just pass a local file and that
 works just fine. There may be something in the URI transformation
 possibly breaking a HDFS source, but try passing a local file - does
 that fail too? The Shell example uses a local file.
 
 On Tue, Aug 6, 2013 at 10:54 AM, Krishna Kishore Bonagiri
 write2kish...@gmail.com wrote:
 Hi Harsh,
 
  Please see if this is useful, I got a stack trace after the error has
 occurred
 
 2013-08-06 00:55:30,559 INFO
 org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set
 to 
 /tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
 =
 file:/tmp/nm-local-dir/usercache/dsadm/appcache

Re: yarn Failed to bind to: 0.0.0.0/0.0.0.0:8080

[Hitesh Shah]

You are probably hitting a clash with the shuffle port. Take a look at 
https://issues.apache.org/jira/browse/MAPREDUCE-5036

-- Hitesh


On Jul 10, 2013, at 8:19 PM, Harsh J wrote:

 Please see yarn-default.xml for the list of options you can tweak:
 http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml.
 The port 8080 is not used by NM by default in the current versions,
 but you can probably find and tweak the right config.
 
 On Thu, Jul 11, 2013 at 7:45 AM, ch huang justlo...@gmail.com wrote:
 i have 3 NM, on the box of one of NM ,the 8080 PORT has already ocuppied by
 tomcat,so i want to change all NM 8080 port to 8090,but problem is
 i do not know 8080 port is control by what option in yarn ,anyone can help??
 
 
 
 -- 
 Harsh J

Re: Compile Just a Subproject

[Hitesh Shah]

Hello Curtis 

Try the following:

hadoop jar ./target/distributed-shell.jar.rebuilt-one 
org.apache.hadoop.yarn.applications.distributedshell.Client -jar ...

If you are running hadoop without the jar command, it will find the first 
instance of Client.class in its classpath which I am guessing is from the jars 
used to start the cluster and not the re-built jar. 

-- Hitesh


On Jun 21, 2013, at 4:09 PM, Curtis Ullerich wrote:

 I should clarify that the build doesn't actually fail (so that bug doesn't 
 seem to be the issue). It reports success. But then the output from the 
 just-build jar doesn't change.
 
 Here's my output from building:
 
 C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshellmvn
  clean package -DskipTests
 [INFO] Scanning for projects...
 [INFO]
 [INFO] 
 
 [INFO] Building hadoop-yarn-applications-distributedshell 3.0.0-SNAPSHOT
 [INFO] 
 
 [INFO]
 [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ 
 hadoop-yarn-applications-distributedshell ---
 [INFO] Deleting 
 C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target
 [INFO]
 [INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ 
 hadoop-yarn-applications-distributedshell ---
 [INFO] Executing tasks
 
 main:
 [mkdir] Created dir: 
 C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target\test-dir
 [INFO] Executed tasks
 [INFO]
 [INFO] --- maven-resources-plugin:2.2:resources (default-resources) @ 
 hadoop-yarn-applications-distributedshell ---
 [INFO] Using default encoding to copy filtered resources.
 [INFO]
 [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ 
 hadoop-yarn-applications-distributedshell ---
 [INFO] Compiling 3 source files to 
 C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target\classes
 [INFO]
 [INFO] --- maven-resources-plugin:2.2:testResources (default-testResources) @ 
 hadoop-yarn-applications-distributedshell ---
 [INFO] Using default encoding to copy filtered resources.
 [INFO]
 [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ 
 hadoop-yarn-applications-distributedshell ---
 [INFO] Compiling 1 source file to 
 C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target\test-classes
 [INFO]
 [INFO] --- maven-jar-plugin:2.3.1:jar (default) @ 
 hadoop-yarn-applications-distributedshell ---
 [INFO] Building jar: 
 C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar
 [INFO]
 [INFO] --- maven-surefire-plugin:2.12.3:test (default-test) @ 
 hadoop-yarn-applications-distributedshell ---
 [INFO] Tests are skipped.
 [INFO]
 [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ 
 hadoop-yarn-applications-distributedshell ---
 [INFO]
 [INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @ 
 hadoop-yarn-applications-distributedshell ---
 [INFO] 
 
 [INFO] BUILD SUCCESS
 [INFO] 
 
 [INFO] Total time: 3.487s
 [INFO] Finished at: Fri Jun 21 16:08:02 PDT 2013
 [INFO] Final Memory: 19M/367M
 [INFO] 
 
 
 
 
 On Fri, Jun 21, 2013 at 3:57 PM, Arpit Agarwal aagar...@hortonworks.com 
 wrote:
 Curtis, if you are running 'mvn clean' in a subproject you may be hitting 
 this bug:
 https://issues.apache.org/jira/browse/HADOOP-9383
 
 You might have better luck if you include install along with package in the 
 same invocation.
 
 -Arpit
 
 
 On Fri, Jun 21, 2013 at 3:12 PM, Curtis Ullerich curtuller...@gmail.com 
 wrote:
 I've executed the commands as you've said, and the jar that is run is 
 unchanged. Here's exactly what I did.
 
 C:\l\ start-dfs
 C:\l\ start-yarn
 C:\l\ cd 
 hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell
 C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell
  hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar 
 target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar 
 -shell_command whoami
 ...output...
 ...modify logging code in 
 C:/l/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
 C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell
  mvn clean install -DskipTests
 

Re: jobtracker webservice in 1.2.0

[Hitesh Shah]

The webservices were introduced only in the 2.x branch. I don't believe the 
feature has been ported back to the 1.x line. If it helps, for the 1.x, line, 
you can try appending ?format=json to the urls used in the jobtracker UI to 
get a dump of the data in json format. 

thanks
-- Hitesh

On Jun 20, 2013, at 8:59 AM, Clay Moody wrote:

 Can someone tell me the REST based web services path for the job tracker web 
 services API, added for v1.2.0
 
 The YARN documentation at 
 (http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html)
   says:
 
 The URIs for the REST-based Web services have the following syntax:
 
   http://{http address of service}/ws/{version}/{resourcepath}
 
 How does this translate to MRv1? I am getting a 404 at 
 http://jobtracker:50030/ws/.…
 
 Thanks
 wcm
 
 
 

Re: YARN Container's App ID

[Hitesh Shah]

Hello Brian,

org.apache.hadoop.yarn.api.ApplicationConstants.Environment should have a list 
of all the information set in the environment. 

One of these is the container ID. ApplicationAttemptID can be obtained from a 
container ID object which in turn can be used to get the App Id. 

-- Hitesh

On Jun 11, 2013, at 11:47 AM, Brian C. Huffman wrote:

 How can a YARN container get its own Application ID?
 
 I tried getting the ApplicationConstants.AM_APP_ATTEMPT_ID_ENV similar to how 
 the Distributed Shell example does for the AppMaster, but that variable 
 doesn't seem to exist in the environment for the container.  Does the App 
 Master have to set it?
 
 Thanks,
 Brian
 

Re: YARN Container's App ID

[Hitesh Shah]

Yes - this is currently on trunk. There were some changes done to make the 
container id available to all containers ( and not only the AM ).

However, for 2.0.3-alpha, I believe you should have access to 
ApplicationConstants.AM_CONTAINER_ID_ENV in the AM's environment. 

-- Hitesh

On Jun 11, 2013, at 12:14 PM, Brian C. Huffman wrote:

 Hitesh,
 
 Is this only in trunk?  I'm currently running 2.0.3-alpha and I don't see it 
 there.  I also don't see it in the latest 2.0.5.
 
 Thanks,
 Brian
 
 On 06/11/2013 02:54 PM, Hitesh Shah wrote:
 Hello Brian,
 
 org.apache.hadoop.yarn.api.ApplicationConstants.Environment should have a 
 list of all the information set in the environment.
 
 One of these is the container ID. ApplicationAttemptID can be obtained from 
 a container ID object which in turn can be used to get the App Id.
 
 -- Hitesh
 
 On Jun 11, 2013, at 11:47 AM, Brian C. Huffman wrote:
 
 How can a YARN container get its own Application ID?
 
 I tried getting the ApplicationConstants.AM_APP_ATTEMPT_ID_ENV similar to 
 how the Distributed Shell example does for the AppMaster, but that variable 
 doesn't seem to exist in the environment for the container.  Does the App 
 Master have to set it?
 
 Thanks,
 Brian
 
 
 

Re: Apache Flume Properties File

[Hitesh Shah]

Hello Raj

BCC-ing user@hadoop and user@hive

Could you please not cross-post questions to multiple mailing lists? 

For questions on hadoop, go to user@hadoop. For questions on hive, please send 
them to the hive mailing list and not the user@hadoop mailing list. Likewise 
for flume. 

thanks
-- Hitesh


On May 24, 2013, at 2:13 PM, Raj Hadoop wrote:

 Hi,
  
 I just installed Apache Flume 1.3.1 and trying to run a small example to 
 test. Can any one suggest me how can I do this? I am going through the 
 documentation right now.
  
 Thanks,
 Raj

Re: getAllocatedContainers() is not returning when ran against 2.0.3-alpha

[Hitesh Shah]

If I understand your question, you are expecting all the containers to be 
allocated in one go? Or are you seeing your application hang because it asked 
for 10 containers but it only received a total of 9 even after repeated calls 
to the RM?  

There is no guarantee that you will be allocated all your requested containers 
in a single allocate() call. The application is expected to keep on making 
allocate calls to the RM at regular intervals and the RM will give it any newly 
allocated containers in the period  since the last allocate call.

The AMRMClient is  a library that should make it easier for someone to write an 
application master. Yes - it is the recommended approach. Please file jiras if 
you see use-cases that are not addressed by it when implementing your AM.

thanks
-- Hitesh

On Apr 3, 2013, at 8:05 AM, Krishna Kishore Bonagiri wrote:

 Hi,
 
  I am running an application that had been written and working well with the 
 hadoop-2.0.0-alpha but when I am running the same against 2.0.3-alpha, the 
 getAllocatedContainers() method called on AMResponse is not returning all the 
 containers allocated sometimes. For example, I request for 10 containers and 
 this method gives me only 9 containers sometimes, and when I looked at the 
 log of Resource Manager, the 10th container is also allocated. It happens 
 only sometimes randomly and works fine all other times. If I send one more 
 request for the remaining container to RM, it could allocate that container. 
 I am running only one application at a time, but 1000s of them one after 
 another.
  
   Is this problem seen by anyone else? I have seen that there are significant 
 number of changes in the way ApplicationMaster.java is written now for the 
 Distributed Shell example. There is now AMRMClient instead of AMRMProtocol. 
 There is method addContainerRequest() which is not there earlier, and there 
 is ConatainerRequest class now which was ResourceRequest earlier, etc.  Is it 
 adivsable to re-write my application in a similar way?
 
 Thanks,
 Kishore
 
 

Re: The most newbie question ever

[Hitesh Shah]

Also, BUILDING.txt can be found at the top level directory of the checked out 
code. 

-- Hitesh 

On Mar 21, 2013, at 5:39 PM, Hitesh Shah wrote:

 Assuming you have checked out the hadoop source code into 
 /home/keithomas/hadoop-common/ , you need to run the maven command in that 
 directory and not from /home/keithomas/   
 
 -- Hitesh 
 
 
 On Mar 21, 2013, at 5:20 PM, Keith Thomas wrote:
 
 Thank you for your help Mayank. I managed to google for BUILDING.TXT and 
 discovered I need maven3. I installed maven3.
 
 When I enter the command on the getting started link you sent me this 
 happens,
 
 keithomas@ubuntu:~$ mvn clean install -DskipTests
 [INFO] Scanning for projects...
 [INFO] 
 
 [INFO] BUILD FAILURE
 [INFO] 
 
 [INFO] Total time: 0.119s
 [INFO] Finished at: Thu Mar 21 17:14:39 PDT 2013
 [INFO] Final Memory: 3M/247M
 [INFO] 
 
 [ERROR] The goal you specified requires a project to execute but there is no 
 POM in this directory (/home/keithomas). Please verify you invoked Maven 
 from the correct directory. - [Help 1]
 
 and when I enter the command you kindly sent this happens,
 
 keithomas@ubuntu:~$ mvn package -Pdist -DskipTests -Dtar
 [INFO] Scanning for projects...
 [INFO] 
 
 [INFO] BUILD FAILURE
 [INFO] 
 
 [INFO] Total time: 0.114s
 [INFO] Finished at: Thu Mar 21 17:15:11 PDT 2013
 [INFO] Final Memory: 3M/247M
 [INFO] 
 
 [WARNING] The requested profile dist could not be activated because it 
 does not exist.
 [ERROR] The goal you specified requires a project to execute but there is no 
 POM in this directory (/home/keithomas). Please verify you invoked Maven 
 from the correct directory. - [Help 1]
 [ERROR] 
 
 
 On Thu, Mar 21, 2013 at 4:38 PM, Mayank Bansal may...@apache.org wrote:
 There is a BUILDING.txt
 
 $ mvn package -Pdist -DskipTests -Dtar
 
 Thanks,
 Mayank
 
 On Thu, Mar 21, 2013 at 4:35 PM, Keith Thomas keith.tho...@gmail.com wrote:
 Thanks for the great link but sorry it looks like this is going to be a long
 process.
 
 Step 1 is: mvn clean install -DskipTests
 
 First of all my OS tells me I need maven but it gives me two choices, maven
 and maven 2.
 
 Is there any pre-req info for Hadoop? I've previously guessed I should
 installed open jdk 6. This time I guess maven 2. The result of the 'clean
 install' is an error,
 
 [ERROR] BUILD ERROR
 [INFO]
 
 [INFO] Cannot execute mojo: clean. It requires a project with an existing
 pom.xml, but the build is not using one.
 
 :(
 
 Looking ahead the next instruction is to CD into a folder that does not
 exist on my machine,
 
 cd hadoop-mapreduce-project
 
 A little after that the instructions note I must have installed version
 2.4.1 or greater. 2.4.1 of what?
 
 
 I feel I am missing the point somewhere :(
 
 
 
 On Thu, Mar 21, 2013 at 4:12 PM, Mayank Bansal may...@apache.org wrote:
 
 http://hadoop.apache.org/docs/r2.0.3-alpha/
 
 On Thu, Mar 21, 2013 at 4:00 PM, Keith Thomas keith.tho...@gmail.com
 wrote:
 After three years in a non-Hadoop Windows wilderness I am lucky enough
 to
 return and wow I find Hadoop 2.0  Brilliant!
 
 I quickly navigated here:
   http://hadoop.apache.org/index.html#Getting+Started
 
 and eagerly downloaded 2.0.3 alpha. Hmmm, now what? I see the following
 sections,
 
 Getting Started
 
 To get started, begin here:
 
 Learn about Hadoop by reading the documentation.
 Download Hadoop from the release page.
 Discuss Hadoop on the mailing list.
 
 
 but I can't find installation instructions. I look for the latest
 version of
 my Tom White collection and the latest book is a year old and is
 unlikely to
 be current enough to be of use to someone as out of date as me. Please
 help
 me to get started. So far I've only managed to download and unzip 2.0.3
 and
 to install open jdk 6.
 
 Many thanks for your patience.
 
 
 
 
 

Re: YARN Features

[Hitesh Shah]

Answers regarding DistributedShell. 

https://issues.apache.org/jira/secure/attachment/12486023/MapReduce_NextGen_Architecture.pdf
 has some details on YARN's architecture.

-- Hitesh

On Mar 12, 2013, at 7:31 AM, Ioan Zeng wrote:

 
 Another point I would like to evaluate is the Distributed Shell example usage.
 Our use case is to start different scripts on a grid. Once a node has
 finished a script a new script has to be started on it. A report about
 the scripts execution has to be provided. in case a node has failed to
 execute a script it should be re-executed on a different node. Some
 scripts are Windows specific other are Unix specific and have to be
 executed on a node with a specific OS.
 

The current implementation of distributed shell is effectively a piece of 
example code to help 
folks write more complex applications. It simply supports launching a script on 
a given number 
of containers ( without accounting for where the containers are assigned ), 
does not handle retries on failures
and simply reports a success/failure based on the no. of failures in running 
the script. 

Based on your use case, it should be easy enough to build on the example code 
to handle the features that 
you require. 

The OS specific resource ask is something which will be need to be addressed in 
YARN. Could you file a JIRA 
for this feature request with some details about your use-case. 


 The question is:
 Would it be feasible to adapt the example Distributed Shell
 application to have the above features?
 If yes how could I run some specific scripts only on a specific OS? Is
 this the ResourceManager responsability? What happens if there is no
 Windows node for example in the grid but in the queue there is a
 Windows script?
 How to re-execute failed scripts? Does it have to be implemented by
 custom code, or is it a built in feature of YARN?
 
 

The way YARN works is slightly different from what you describe above. 

What you would do is write some form of a controller which in YARN terminology 
is referred to as an ApplicationMaster. 
It would request containers from the RM ( for example, 5 containers on WinOS, 5 
on Linux with 1 GB each of RAM ). Once, the container is
assigned, the controller would be responsible for launching the correct script 
based on the container allocated. The RM would be responsible
for ensuring the correct set of containers are allocated to the container based 
on resource usage limits, priorities, etc. [ Again to clarify, OS type 
scheduling is currently not supported ]. If a script fails, the container's 
exit code and completion status would be fed back to the controller which 
would then have to handle retries ( may require asking the RM for a new 
container ).



 Thank you in advance for your support,
 Ioan Zeng

Re: YARN Features

[Hitesh Shah]

Answers inline. 

-- Hitesh

On Mar 12, 2013, at 12:26 PM, Ioan Zeng wrote:

 Another evaluation criteria was the community support of the framework
 which I rate now as very good :)
 
 I would like to ask other questions:
 
 I have seen YARN or MR used only in the context of HDFS. Would it be
 possible to keep all YARN features without using it in relation with
 HDFS (with no HDFS installed)?

It uses the generic filesystem apis from hadoop to a very large extent so it 
should work with any filesytem solution. 
There are a couple of features which do depend on HDFS though - log aggregation 
for example ( collect all logs of all containers into a
central place ) that would need to be disabled. There may be some cases which I 
am may be unaware of. If you do see anything which 
depends on HDFS, please do file jiras so that we can address the issue.

 
 You mentioned the CapacityScheduler. Does this require MapReduce? or
 is it included in YARN? I understood that MRv2 is just an application
 built over the YARN framework. For our use case we don't need MR.
 

Yes - you are right - there would be no dependency on MapReduce. 
The CapacityScheduler is the scheduling module used inside the ResourceManager 
( which is YARN only ). 


 For a better understanding of my questions regarding the Distributed
 Shell. We intend to use YARN for a distributed automated test
 environment which will execute set of test suites for specific builds
 in parallel. Do you know about similar usages of YARN or MR, maybe
 case studies?
 

There are a few others who are using Yarn in various scenarios - none who use 
it for their test infrastructure as far as I know. 
The closest I can think of would be LinkedIn's use-case where they launch and 
monitor a bunch of services on a Yarn cluster.  
( http://riccomini.name/posts/hadoop/2012-10-12-hortonworks-yarn-meetup/ might 
be of help )


 Thanks,
 Ioan
 
 On Tue, Mar 12, 2013 at 8:47 PM, Hitesh Shah hit...@hortonworks.com wrote:
 Answers regarding DistributedShell.
 
 https://issues.apache.org/jira/secure/attachment/12486023/MapReduce_NextGen_Architecture.pdf
  has some details on YARN's architecture.
 
 -- Hitesh
 
 On Mar 12, 2013, at 7:31 AM, Ioan Zeng wrote:
 
 
 Another point I would like to evaluate is the Distributed Shell example 
 usage.
 Our use case is to start different scripts on a grid. Once a node has
 finished a script a new script has to be started on it. A report about
 the scripts execution has to be provided. in case a node has failed to
 execute a script it should be re-executed on a different node. Some
 scripts are Windows specific other are Unix specific and have to be
 executed on a node with a specific OS.
 
 
 The current implementation of distributed shell is effectively a piece of 
 example code to help
 folks write more complex applications. It simply supports launching a script 
 on a given number
 of containers ( without accounting for where the containers are assigned ), 
 does not handle retries on failures
 and simply reports a success/failure based on the no. of failures in running 
 the script.
 
 Based on your use case, it should be easy enough to build on the example 
 code to handle the features that
 you require.
 
 The OS specific resource ask is something which will be need to be addressed 
 in YARN. Could you file a JIRA
 for this feature request with some details about your use-case.
 
 
 The question is:
 Would it be feasible to adapt the example Distributed Shell
 application to have the above features?
 If yes how could I run some specific scripts only on a specific OS? Is
 this the ResourceManager responsability? What happens if there is no
 Windows node for example in the grid but in the queue there is a
 Windows script?
 How to re-execute failed scripts? Does it have to be implemented by
 custom code, or is it a built in feature of YARN?
 
 
 
 The way YARN works is slightly different from what you describe above.
 
 What you would do is write some form of a controller which in YARN 
 terminology is referred to as an ApplicationMaster.
 It would request containers from the RM ( for example, 5 containers on 
 WinOS, 5 on Linux with 1 GB each of RAM ). Once, the container is
 assigned, the controller would be responsible for launching the correct 
 script based on the container allocated. The RM would be responsible
 for ensuring the correct set of containers are allocated to the container 
 based on resource usage limits, priorities, etc. [ Again to clarify, OS type
 scheduling is currently not supported ]. If a script fails, the container's 
 exit code and completion status would be fed back to the controller which
 would then have to handle retries ( may require asking the RM for a new 
 container ).
 
 
 
 Thank you in advance for your support,
 Ioan Zeng
 

Re: Installing Hadoop on RHEL 6.2

[Hitesh Shah]

You could try using Ambari. 

http://incubator.apache.org/ambari/
http://incubator.apache.org/ambari/1.2.0/installing-hadoop-using-ambari/content/index.html

-- Hitesh

On Feb 13, 2013, at 11:00 AM, Shah, Rahul1 wrote:

 Hi,
  
 Can someone help me with installation of Hadoop on cluster with RHEL 6.2 OS. 
 Any link or steps will be appreciated to get me going. Thanks
  
 -Rahul
  

Re: “hadoop namenode -format” formats wrong directory

[Hitesh Shah]

Try running the command using hadoop --config /etc/hadoop/conf to make sure 
it is looking at the right conf dir.  

It would help to understand how you installed hadoop - local build/rpm, etc .. 
to figure out which config dir is being looked at by default.

-- Hitesh

On Feb 6, 2013, at 7:25 AM, Andrey V. Romanchev wrote:

 Hello Samir
 Yes. Here is full section from my hdfs-site.xml
 
   property
 namedfs.name.dir/name
 value/mnt/ext/hadoop/hdfs/namenode/value
 description
 Determines where on the local filesystem the DFS name node
 should store the name table.  If this is a comma-delimited list
 of directories then the name table is replicated in all of the
 directories, for redundancy.
 /description
 finaltrue/final
   /property
 
 
 2013/2/6 Samir Ahmic ahmic.sa...@gmail.com
 Hi, Andrey
 
 Did you try finaltrue/final in your config ? It should be like this:
 
 namedfs.name.dir/name
 value/mnt/ext/hadoop/hdfs/namenode/value
 finaltrue/final
 
 Cheers
 
 
 On Wed, Feb 6, 2013 at 4:07 PM, Andrey V. Romanchev
 andrey.romanc...@gmail.com wrote:
 
  Hello!
 
  I'm trying to install Hadoop 1.1.2.21 on CentOS 6.3.
 
  I've configured dfs.name.dir in /etc/hadoop/conf/hdfs-site.xml file
 
  namedfs.name.dir/name
  value/mnt/ext/hadoop/hdfs/namenode/value
 
  But when I run hadoop namenode -format command, it formats 
  /tmp/hadoop-hadoop/dfs/name directory instead. It seems like default 
  directory.
 
  What am I missing?
 

Re: hortonworks install fail

[Hitesh Shah]

Hi 

ambari-user@ is probably the better list for this. 

It seems like your puppet command is timing out. Could you reply back with the 
contents of the /var/log/puppet_apply.log from the node in question? 

Also, it might be worth waiting a few days for the next release of ambari which 
should be a lot more stable than the previous one.

thanks
-- Hitesh

On Jan 9, 2013, at 1:33 AM, cldo cldo wrote:

 I install hortonworks.When i deployed it had a error:
 “nodeReport”: {
 “PUPPET_KICK_FAILED”: [],
 “PUPPET_OPERATION_FAILED”: [
 hserver1
 ],
 “PUPPET_OPERATION_TIMEDOUT”: [
 hserver1
 ],
 “PUPPET_OPERATION_SUCCEEDED”: []
 },
 “nodeLogs”: []
 System check results:
 6. Firewall
 iptables: Firewall is not running.
 
 7. SELINUX
 SELinux status: disabled
 
 8. NTP
 ntpd (pid  7574) is running...
 
 9. Name resolution
 Server: 127.0.0.1
 Address:127.0.0.1#53
 
 Name:   hserver1
 Address: 192.168.3.147
 
 root ssh localhost passwordless success.
 Thanks
 
 

Re: Hive error when loading csv data.

[Hitesh Shah]

Michael's suggestion was to change your data to:

c|zxy|xyz
d|abc,def|abcd

and then use | as the delimiter. 

-- Hitesh

On Jun 26, 2012, at 2:30 PM, Sandeep Reddy P wrote:

 Thanks for the reply.
 I didnt get that Michael. My f2 should be abc,def
 
 On Tue, Jun 26, 2012 at 4:00 PM, Michael Segel 
 michael_se...@hotmail.comwrote:
 
 Alternatively you could write a simple script to convert the csv to a pipe
 delimited file so that abc,def will be abc,def.
 
 On Jun 26, 2012, at 2:51 PM, Harsh J wrote:
 
 Hive's delimited-fields-format record reader does not handle quoted
 text that carry the same delimiter within them. Excel supports such
 records, so it reads it fine.
 
 You will need to create your table with a custom InputFormat class
 that can handle this (Try using OpenCSV readers, they support this),
 instead of relying on Hive to do this for you. If you're successful in
 your approach, please also consider contributing something back to
 Hive/Pig to help others.
 
 On Wed, Jun 27, 2012 at 12:37 AM, Sandeep Reddy P
 sandeepreddy.3...@gmail.com wrote:
 
 
 Hi all,
 I have a csv file with 46 columns but i'm getting error when i do some
 analysis on that data type. For simplification i have taken 3 columns
 and
 now my csv is like
 c,zxy,xyz
 d,abc,def,abcd
 
 i have created table for this data using,
 hive create table test3(
 f1 string,
 f2 string,
 f3 string)
 row format delimited
 fields terminated by ,;
 OK
 Time taken: 0.143 seconds
 hive load data local inpath '/home/training/a.csv'
 into table test3;
 Copying data from file:/home/training/a.csv
 Copying file: file:/home/training/a.csv
 Loading data to table default.test3
 OK
 Time taken: 0.276 seconds
 hive select * from test3;
 OK
 c   zxy xyz
 d   abcdef
 Time taken: 0.156 seconds
 
 When i do select f2 from test3;
 my results are,
 OK
 zxy
 abc
 but this should be abc,def
 When i open the same csv file with Microsoft Excel i got abc,def
 How should i solve this error??
 
 
 
 --
 Thanks,
 sandeep
 
 --
 
 
 
 
 
 
 --
 Harsh J
 
 
 
 
 
 -- 
 Thanks,
 sandeep

Re: Running Distributed shell in hadoop0.23

[Hitesh Shah]

The shell script is invoked within the context of a container launched by the 
NodeManager. If you are creating a directory using a relative path, it will be 
created relative of the container's working directory and cleaned up when the 
container completes. 

If you really want to see some output, one option could be to have your script 
create some data on hdfs or echo output to stdout which will be captured in the 
container logs.  The stdout/stderr logs generated by your script should be 
available wherever you have configured the node-manager's log dirs to point to. 

-- Hitesh 

On Dec 14, 2011, at 10:52 PM, raghavendhra rahul wrote:

 When we create a directory using distributed shell,any idea where it is 
 created
 
 On Thu, Dec 15, 2011 at 11:57 AM, raghavendhra rahul 
 raghavendhrara...@gmail.com wrote:
 How to run any script using this.When i tried it shows final status as failed.
 
 
 On Thu, Dec 15, 2011 at 11:48 AM, raghavendhra rahul 
 raghavendhrara...@gmail.com wrote:
 Thanks for the help i made a mistake of creating symlinks within modules.Now 
 everythng is fine.
 
 
 
 On Thu, Dec 15, 2011 at 11:18 AM, raghavendhra rahul 
 raghavendhrara...@gmail.com wrote:
 should i link the hadoop-yarn-applications-distributedshell-0.23.0.jar also.
 Without linking this jar it throws the same error.
 If linked it shows
 at org.apache.hadoop.util.RunJar.main(RunJar.java:130)
 Caused by: java.util.zip.ZipException: error in opening zip file
 at java.util.zip.ZipFile.open(Native Method)
 at java.util.zip.ZipFile.init(ZipFile.java:131)
 at java.util.jar.JarFile.init(JarFile.java:150)
 at java.util.jar.JarFile.init(JarFile.java:87)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:128)
 
 
 
 
 

Re: Running Distributed shell in hadoop0.23

[Hitesh Shah]

Assuming you have a non-secure cluster setup ( the code does not handle 
security properly yet ), the following command would run the ls command on 5 
allocated containers.
 
$HADOOP_COMMON_HOME/bin/hadoop jar  path to 
hadoop-yarn-applications-distributedshell-0.24.0-SNAPSHOT.jar 
org.apache.hadoop.yarn.applications.distributedshell.Client --jar  path to 
hadoop-yarn-applications-distributedshell-0.24.0-SNAPSHOT.jar --shell_command 
ls --num_containers 5 --debug 

What the above does is upload the jar that contains the AppMaster class to 
hdfs, submits a new application request to launch the distributed shell app 
master on a container which then in turn runs the shell command on the no. of 
containers specified. 

-- Hitesh

On Dec 14, 2011, at 1:06 AM, sri ram wrote:

 Hi,
  Can anyone give the procedure about how to run Distibuted shell example 
 in hadoop yarn.So that i try to understand how applicatin master really works.

Re: Running Distributed shell in hadoop0.23

[Hitesh Shah]

The yarn jars are likely missing from the classpath. Could you try creating the 
symlinks as per step 11 from 
http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-mapreduce-project/INSTALL?revision=1166955
 ?

-- Hitesh

On Dec 14, 2011, at 8:35 PM, raghavendhra rahul wrote:

 I get the following erroer by the given command to run distributed shell
 hadoop1@master:~/hadoop/bin$ ./hadoop jar 
 ../modules/hadoop-yarn-applications-distributedshell-0.23.0.jar 
 org.apache.hadoop.yarn.applications.distributedshell.Client --jar 
 ../modules/hadoop-yarn-applications-distributedshell-0.23.0.jar 
 --shell_command ls --num_containers 5 --debug
 2011-12-15 10:04:41,605 FATAL distributedshell.Client (Client.java:main(190)) 
 - Error running CLient
 java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/ipc/YarnRPC
 at 
 org.apache.hadoop.yarn.applications.distributedshell.Client.init(Client.java:206)
 at 
 org.apache.hadoop.yarn.applications.distributedshell.Client.main(Client.java:182)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:616)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:189)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.yarn.ipc.YarnRPC
 at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
 ... 7 more
 
 
 On Thu, Dec 15, 2011 at 12:09 AM, Hitesh Shah hit...@hortonworks.com wrote:
 Assuming you have a non-secure cluster setup ( the code does not handle 
 security properly yet ), the following command would run the ls command on 5 
 allocated containers.
 
 $HADOOP_COMMON_HOME/bin/hadoop jar  path to 
 hadoop-yarn-applications-distributedshell-0.24.0-SNAPSHOT.jar 
 org.apache.hadoop.yarn.applications.distributedshell.Client --jar  path to 
 hadoop-yarn-applications-distributedshell-0.24.0-SNAPSHOT.jar 
 --shell_command ls --num_containers 5 --debug
 
 What the above does is upload the jar that contains the AppMaster class to 
 hdfs, submits a new application request to launch the distributed shell app 
 master on a container which then in turn runs the shell command on the no. of 
 containers specified.
 
 -- Hitesh
 
 On Dec 14, 2011, at 1:06 AM, sri ram wrote:
 
  Hi,
   Can anyone give the procedure about how to run Distibuted shell 
  example in hadoop yarn.So that i try to understand how applicatin master 
  really works.